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ABSTRACT c . 

A project for studying ways to optimize utilization 
of evaluation produces in public schools is reported. The results 
iriHicate that the negative picture of use prevalent in recent 
literatute sterns from the unrealistic expectation that local 
decision-makers will behave jln a classically rational manner. Such a 
view ignores the political settings of real world evaluation use 
where the non-use of results may constituted viable and "rational" 
alternative. Empirical research has suggested that results are used 
and in a variety of ways. Factors that have not been demonstrated to 
affect use include the methodological quality of the evaluitions^and 
the timeliness of its report. Factors that do seem to effect use^\ 
include the report source, content, and reteiver; political 
considerations; ahd'the "personal f actor, " i .e. the presence of at 
least one person who cares about the evaluation and its results. A 
collabprative approach to evaluation capitalizes, on the factors that 
have been shown to affect use, although the practicability of such an 
approach at*the local level may make it unfeasible. (Author/GK) 
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This paper i& the final report of a 1980-1981 NIE grant to 

' > * I ' 

Orleans Parish School Board (NIE-G-80-0082) for .studying ways 



to optimize the utilization of ^valuation prpducts in public 
schools. Host of ( the literature discussed here was compiled .frojm 
JuW to Novembe^ 1980* Our compilation was* aided greatly by two 
earlj^r bibliographies prepared at Northwest Regional Educational 
^Laborator^ '(Smith, 1980a; Hansen ajidvMartin'/ 1980) . . In addition, 
we conducted an ERIC computer search and reviewed the Eddcation 

0 * 

Index evaluation listings* from 1975^>n and the .American 
Educational Research Association's Annual Meeting program ,and 
I abstracts for 1978, 1979, and 1980. The resulting bibliographies 
are available through ERIC in two' forms — an extensive 
bibliography o*f 326 entries (King, Thompson, &. Pechman, 1981a) . 
and a shorter bibliography annotating roughly 20 important pieces 
(King, Thompson, & Pechman, J.981b) . To be* included, papers, 
articles, chapters, and books had to meet two criteria: 

1. They either had to address directly the topid of the 
utilization of program evaluation* informat ion at the locfel 
education agency (LEA) level and/or have been extensively^ci ted 
in the literature and hence be important conceptually . This ' 
eliminated much of ithe body of literature on policy analysis and 
evaluation use at the* federal leyel, as. well as literature 
related to other evaluation settings (e.£. ,* public health, social 
work, hospitals). This also eliminated work concerning the 
of research results or of knowledge in general and on. the 

\ ■ ' ■ > 
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evaluation of teaching performance. 



2^ The 



literature had to be relatively current and readi 



iy 



available. Anything ten years or older was' included only when it 
was seminal or had clearly influenced later thinking in the 



• field. Pieces that we were unable to locate after extensive 

> 

• checking were eliminated. * «> 

Any bibliography, in a rapidly chan^ng field like 

^-evaluation must be r/egarded as dated even before it appears. For 

/ A 

this r*eason, some ofjthe literature discussed in the following • 

/ ■ ' 

paper is not citedjin the earlier bibliographies, and .readers are 

additionally encouraged to use the bibliography at the end of * 

this Raper in their examination of the evaluation use literature. 

^ For^ the purpose of this paper, the term evaluation has been 

broadly defined. The question of definition is an important one 

i 

since what one counts as evaluation information is clearly 

^ > , . - V 

affected by on"e s definition of evaluation. Webster and 

\ 

Stufflebeam (1978; see also Stufflebeam and Webster, 1980) 
identify 13 alternative approaches to evaluation, grouped in -the 
three categories of pseudo-evaluation, quasi*evaluat i<5n, andytjue 
evaluation. Given these, categories/ the .evaluation ' process car? 
involve radically different activities, and "evaluat idn" results 
can range from a carefully manipulated public relations campaign 
to v an empirical generalization published in' an educational 
journal; from, a'hational policy to a detailed cas£< study; fronTa 
staff brain-storming - session to the administration of a test. 

In addition ta this categorization are numerous others in 
the field, for example, the distinctions between formative and 



summative evaluation (Scriven, 1967; v Alk : in et al., 1974) and 

among the research, decision-making, arid values^ assessment 

comjxments of evaluation {Smith^ ^80c) • It' is beyond the scope 

* _ 

of this paper to discuss the implications of the various 

» . ' 

approaches and components of Evaluation. For this reason, the 

following! definition of evaluation is stipulated: "the process 

of providing information to administrators* to help dhem make 

decisions regarding educational programs." This definition is 

4 

purposely broad, encompassing pseudo-, quasi-, and true 

evaluation. " Such a definition is needed to discuss evaluation 

use ih local settings where intuitive definitions -abound, "often 

^equating evaluation with achievem ent test ing (David, 1981K To 

limit the definition to Webster and Stuff lebeam's notion of tru^ 

evaluation is *tp lose sight* of many examples of use. The 

decision-making focus of , the definition relates to this project's 

purpose, i.e. to determine Strategies for increasing evaluation 

use by LEA personnels Following a conceptual discussion, the 

paper is divided into a summary of empirical work -and a 

> 

discussion of other writings on ways to increase the use of 
evaluation information. 

Several names can apply to local users of evaluation 
information. Throughout the paper, the terms decision - maker , 
administrator , and information user . are used interchangeably .\ 
Also, following Weiss (1?79), , the term) use,* rather than ) 
utilization ,' will be used. As Weiss writes, utilization 
"embodies an inappropriate imagery . . ** because' of its overtones 
of instrumental episodic application. People don't utilize 



« ■ * 

research the way that they utilize a hammer" (p._2) . •* 

The authors wish to thank Robert Brown, James Sanders, and 
Michael Vitale, who wrote critiques of a draft of this paper, for 
their valuable and insightful comments. • 
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c - OPTIMIZING EVALUATION USE ■ 

(NIE-G-80-0082) - ^ ^ 
, Abstract , / 

The purpose of the project was to facilitate optimal use of evaluation 
research by generating knowledge regarding those factors ^that affect utilization. 

i 

The project has resulted in three products: (1) a compilation of previous 
theorizing and research related tcMbcal evaluation use, including an extensive 
bibliography, an annotated bibliography, and an ^integrated literature review; 
(20 a survey of local administrators 1 perceptions of evaluations^ and (3) a 
report simulation study of local users 1 perceptions of evaluetorsi 

i 

The results indicate that the negative picture, of use prevalent in recent 
literature steins from the unrealistic expectation that local decision-makers 
will behave in a classically rational manner. Such a view ignores the political 
settings of real world evaluation use where the non- use of results may consti- 
tute a viable and "rational 11 alternative. Empirical research has suggested 
that results are used and in a variety of ways.' Factors that have not been 
demonstrated to affect use include the methodological quality* of the evalua- 
tions and the timeliness of its report. Factors that do seep to affect use 
include the report source, content, and receiver; political considerations; 
apd the "personal factor," i.e. the presence of at least one person who cares 
about the evaluation and its results. A collaborative approach to evaluation 
'gapltalizes on the factors that have been shown to affect use, although the 
practicability of such an approach at the local level may make it unfeasible. 

/ 
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3CNIE-G-8O-0O82) 
Executive Summary , 1 

; . • 

For the past ten years the literature on evaluation use has lamented the 
fact that, despite millions of dollars being spent on educational evaluati 
the results were not being used by local decision-makers. Such non-use is 
costly because it wastes effort and money, but more importantly because it may* 
mean* children are being served by less than optimally effective programs. Our 
project sought to riq, three things: (1) ; to compile and integrate the extant 
literature on local evaluation use; * (2/ to survey administrators nationwide 
as to t^eir feelings about ev&l^ation; and (3) to conduct a report simulation 
study t» See if local administrators perceived differences among eValuators. 

The projects first product is a state-of-th$-art paper entitled "Local 
Evaluation Use: A Literature Review and Research Agenda" (King and Thompson, 
1981b). The paper makes several major points. First* decisionmaking in the f 
real world of public schools is not a rational process in the traditional sense. 
Local users of evaluation information engage in "satisf icing," i.'e. in finding 
solutions that are "good enough" given constantly changing and uncertain cir- 
cumstances. Second, and related to this, is the fact that the non-use of evalu- 
ation ^results must not be viewed as irrational or non-rational; in an important 
sense, non-use represents ,a viable form of use in* the political contexts of 
local evaluation settings. Third, (then, the concept of use must be broadened 
from the limited kbtion of instrumental use (i.e. where a result suggests a % 
Specific action and that action -is taken) to include & variety of activities. 
A necessary condition for use is the conscious employment of results by a 
decision-maker for whatever purpose; without such awareness and choice on some- 
one s part, the activity that ensues is not use, but rather, to use Spith's. 
(1980) term, evaluation impact. Theorists have generated several frameworks 'a 
for thinking about evaluation use, supporting the broadened definition. 

Fourth, 'few empirical studies of ]»cal evaluation use can be found in the 
lifei^ture, Most writing that has been done on, the topic is informal and anec- * ^ 
dotpl. 'The research that has been done has centered around three questions: 
(1) Are evaluation results used? * (2) What are;the characteristics of local 
evaluation units and useps? and O^What fatten* affect use at the local 'level? \ * 
The answer to the'first question is def initely'yes, if one adopts the broadened „ ' 
definition of use discussed above. Local administrators do use evaluation < 
results, although not necessarily in predictable ^ays. . * 1 

The answers to the secopdQand third questions are far more complex. Exist- 
ing research has only begun to provide insight* into the process of use at the 
local level. Regarding the second question, two generalizations oan be made: 
first, local evaluatioh units are typically not involved in program improvement; « T 
and second, although some research indicates that local users find evaluation 
helpful, other 'data suggest that these users may feel that evaluation^ not 
of value to them. The picture of -the local evaluation context currently avail- 
able suggests reason f6r the non-use of evaluation information. V>* 



i 



-The factors affecting iise at a -local levgl ca^ be> divided into those 
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that evaluators control and those that are part of the evaluation context" and 
must therefore be worked with./ Research suggests that methodological quality 
does not necessarily lead to /he use of results, although good evaluation prac- 
tice nevertheless stresses its importance. Variations in evaluation reports 
have been shown to affect use; the message source, content, and the characteris- 
tics of the receiver can all affect what happens to results. Interestingly, the 
variable of timeliness., often cited as a critical factor -for use, has not been 
shown to be significant, probably because of the informal contacts between 
evaluators and clients that ^ake .the evaluation report a mere formality. • 

•Research has also pointed to the importance of certain environmental and 
contextual factors affecting use. The issues faced by various user*? differ 

1 S pS^iT 7 ' 3n Ai the V3lue ° f evaluation information for addressing some prob- 
lems (e g rapidly changing issoies at the district^level) ■ is", to date, 
unclear (Kennedy, Apling, and Neumann^ 1 980.) . Two empirically validated fac- 
tors are the political considerations factor" and the "personal factor," * 
trV,- P J e r ^ 0j \ 3t i eaSt ° ne person in an valuation who cares about the 
rlT.Tr \ aT tt itS f . reSults (Patton « 1978). Given these issued and factors, the 
research .literature, repeatedly recommends a collaborative approach to evalua- ' 
tion, despite the fact that such an approach may.be extremely time-consuming 
and has not been empirically dempnstrated to be effective * 



bv *J I*™? v, P ! 8 5 ?f? er makeS are based on # he anecdotal literature that 
1J ,} g f ° Und valida£ion in empirical wofT, Suggestions for improving 

writ?™ l*lTi rtS 1181118 multi " media presSlgwS,- educating the press", 

writing executive summaries, including carefully wooded policy recommendations 
tTrLlL* ^ importance of informal contacts between evaluators and clients 

!2T! - ? 3necd ° tal literature^suggests that the formal reportr itself Tay be- 
iTolltTul?:* ?> ; tk t '° llaboratlVe ^Proach finds support despite inherent 
half Z ^ 80318 ? huJ ? le * whereb y *cal staff decide to switch goals 

rill ItLl f f? ? 3n evaluation > 0r the "methodology shuffle," whereby users 
rent statisticians- to attack ari^ results that are unpopular.* 

The non-empirical evaluation literature suggests numerous approachesvfor ' 
I3w£? G °^ bo f at 4 ion - Recommendations include the following; targeting 
evaluations, identifying issues of interest to users, employing^ ormal report- 
ing procedures, anticipating users' future questions, anS showing users thaE 
the evaluator cares. A further approach of particular merit for local users V 
which rlptprm 3 called evaluability assessment (see, ^ example, Rutman," 1980) 
which. determines when and if a program is ..ready to'bTevaluated. 

* * * 

• The final point ma de in the paper; is an obvious ' one: we have much to learn 
SfL^" 8 ? evaluation information at the local level. 'Previous research - 
J° mina 5 ed b y retrospective case studies/and simulation studies, both . • 
o£ which have limitations -as well as strengths. Research is needed fn- several 
directions, both conceptual and practical. Most needed, perhaps, is -prospective 
research on the process of evaluation ,use, i.e. current .case studies, and - 
research on the outcomes of effectiveness training for evaluators. 

* 

Because they were empirical studies, the-sscoftd and third project products' ' 

Sni'JTSS^ m To^ lf \ nationwide su rvey of local administrators . 
(King and Thompson, 1981a) found first, that the 'majority of ysers surveyed- 

IZTlT.T '5*? ^ .^alua"ons of educational programs in their schools 

were either useful or vefy useful. Second, regardless of district size^most k 



•us>ers Reported tl^at they do use the information - they recfeive in their decision- 
making. The survey alio revealed/two less heartening results i well. ovexf two- 
th±r<fs of those reporting (72%) felt that the program effects they mpst crayed 
about could only be measured indirectly (42^5) "or not at all (30%); '.ami fdto 
local users hsfve much contact with evaluators." % Given these differing perspec- 
tives on measurement and^limited contact between evaluators anci users, the 
positive attitude and use of results mentioned earlier may *e indicative of 
evaluation 1 s *power, ^ % # 

-The thJLx^d product o^ the project was a study entitled "Evaluation Types: 
po Differences 'Matter?^ (Thompson and King, 1981). .^he study investigated 

* whether administrators implicitly perceive the four types of evaluators prcf- 
ppsfed< by Meltsner (1976) (I.e. entrepreneur, politician, technician, and place- 
fil&ldfer) when they consider evaluative information. One simulated evaluation 

Report yas written to represent each eyaluator type, then two pilot studies 
\gere •conducted' to §ssess .the Validity df the simulation materials. . Tho, results 
suggest' that administrators may implicitly "employ the typology; they were par- 
ticularly attentive to the technical merit of reports. 
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Introduction 



Reading the- literature on the use of .program evaluation 
inf^m^tion may discourage e^n 'stout-hearted ev^luatbrjsu * A 
sampl-ing of quogptions §rom the last ten years is sufficient to 

document the widespread feeling among evaluation commentators 

• H , \ 

that program evaluation results in gen'eral are not having' the 

♦ * ' a * . 

-effect they were intended to have, Wholey et al. (1970) 

concluded that "the recent literature is unanimous in announcing 

ithe general '.failure of evaluation- to affect decision-making in a 

significant way" (p. 46). In a T972 paper on evaluation * 

utilization, Weiss wrote that "evaluators complain about 'many 

things, but their most common complaint is that their ' findings 

are ignored" (p. 319). Writing a year later, Worthen and Sanders 

(1973J^&otea that "evaluation --is one of the mast widely discussed 

little used processes in today's* educational systems" (p. 1) . 

The same year, Rippey (1973) concluded 'that > 

At the/moment, there seems to* be no evidence that 
evaluation although the, law of the land, contributes 
anything to educational practice other than headaches 
for the researcher, , threats for the innovators, and 
depressing articles for journals devoted to evaluation 

Five years'later Pat'ton (1978) wrote that * 

In many ways the odds are all a,gainst' utilization and 
it is quite possible to become, skeptical about the 



futility of trying to have ,impact in a world where 
situation after s'ituat: 
(p. 291). 



ion seems impervious Jto change 



>s^tecently, Alkin and Daillak (1979) stated tha£ "there have been 
' v great hopes for evaluation, not "only among evaluators themselves, 
but also among ottier, educators, elefcted officials, and. the -public. 
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Yet these hopes have dimmed" (p. 41). The uneasy feeling that 
evaluation results are not being used has not changed. As Stake 
(1976) once wirote, "We do not know whether or not evaluation is ' . 

r - I 

going< to contribute more to the problems of education or more to 
^ the solutions" (p. 1) . % 

This characteriteation apparently applies equally well to 
judicial (Saks, 1980), and legislative setting^ (>Brandl, 1980'; 

7 

Mitchell, 1980) , and r unfortunately, to local education agencies as 

well. Within > the last two years, three writers involved in 

. pqblic school-settings have written the following: 

In an ideal world we wouldn't have to wgrry about 
v utilization. Educators would be' eagerly' awaiting- our 

♦ findings and would promptly rush to put them into 
" ' practice^ I don't need to tell you that isn't 
happening (Holley, 19^9, p. 2) . * 

. . .although the potentiality exists, local us§ of 
evaluation . . . does not occur routinely as a natural 

* e consequence of conducting an -evaluation- (Kennedy, 

/ Apling, & Neumann, 1980, p. 5) . 

AIl.LEAs, with possibly a few exceptions, can point to 
their volumes of research and evaluation verbiage 
sitting on the shelves of district administrators 
* being used for little else than a door stop, swatting 

flies, or-any of the other various and sundry purposes * 
for which research is used in the public schools 
(Kilbourne & DeGracie, 1979, p. 12).. 

'The consensus ^in the literature is virtually unanimous: the 

results of program evaluations are having little effect in the v 

very places they were meant to-i 

•Although these commentators are expressing largely 

undocumented personal feelings, the concern is a valid one. Alkln 

et-al. (1974) have written that "if evaluative information is not 

useful, if it does, not serve the needs of decision makers, tfien r _ 



/ 
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evaluation has lost its justification" (p. 1) . To the extent that 
evaluative results that could have a positive effect go unused, 

the entire evaluation enterprise in some sense* fails. * 

* * * 

The non-use of evaluation findings when that. use would be ' 

appropriate is costly in three ways* ^irst, non-use represents an 

enormous' waste , of effort. As Datta (19*79) notes, • v " 

Considerable ef fort is 'involved ' in conducting almost 
' * any evaluation: in identifying the evaluation 

^que'stion, in designing the study, >ip overcoming the 
obstacles to conducting an evaluation and protecting 
it methodologically from uninterpretability (p. 22) 

Wt k Second, non-use represents a* waste of monies. At the 

federal level,, for example, direct expenditures on non-defense 

evaluation projects in 1974 by the federal go\fernment alon'e, - 

amounted to $146 millfon (Kelezo, l$Tpf[ Surprisingly, at the" , ' 

very time theorists and users alike. are questioning evaluations'^ 

ST 

results- and effects, money continues to be invested in evalu^4fion 

fc * * ■ 

studies.- Th§ indirect costs of non-use', when ineffective 
programs are not modified or discontinued, are even more 
stagger ing* at both the . loca,l and the federal levels. 

The third and most serious costN^f non-use,* however, is the 
human cost: non-use means that the clients of educatio'nal 
programs receive less than opt imally* effect ive service. Failure 
to use evaluative information is ^speciajly unfortunate because, 
as Wise (1980) notes, "No one else is given the resources and 
time 'to question, observe, assess, weigh, probe, and ief lect that 
the evaluator is given" (p% 16) . At the local level £ the costs 
of non-use can weigh ^heavily. * a 

This paper is a review and integration of the^JLi tera^ure on 
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evaluation utilization, with special reference to the use of 
evaluation findings at the local level. Following a discussion 
of the rationality of us.e and non-use, the^aper will present a 
conceptual framework for evaluation use, a review of empirical 
research on the topic, and a discussion of observations and 
recommendations -from the literature. The final section of the 

j * 

paper will include a critique of previous research and a research 

agenda for further investigation of local evaluation use. 

*- > 
" The Rationality of Use and Non-Use 

: < 

Well before the passage of the Elementary and Secondary 
t Education Act in 1965 signalled a new era'for program evaluation, 
t educators paid homage, to a myth concerning the presumed place of 
•evaluation' in the world of education. The myth went something * 
like this: ^ 1 

Harried decision-makers, over-worked but rational at 
heart, are presented evaluative information tha^-is 
relevant to decisions they are about to make, fitting 
9 calmly at their desks, they consider the data, weigh 
; ^ their options, then make the "correct" decision, i.e. ' ' 
the choj.ce supported by the data. The evaluator, 
beaming happily behind the scenes, takes pride in 
nurturing "such rational action. ' 

The series of quotations given at the beginning of this paper 
suggests that this jnyth--what Andrews (1979) calls the "big bang" 
theory of evaluation— has rightfully been exploded. First,, such 
a view denies the process and political context of real world 
decision-making, and second* kt ignores the rationality , of 
nog-use. % 

becisign-making in the Real World . Perceptions of how 
decisions are made clearly effect judgments of the extent and 



quality- of the use of evaluative information. ' As Wise (1978) 

explains, % ... 

Referring to administrators as "decision-makers" and • 
to what they do as "decision-making* may have been a 
,f irst step in creating the utilization problem, for we 
expect to see decisions being made by someone called a 
decision-make** {p.' 6) ♦* 

?bus, many evaluators presume that evaluation ought to be used 

merely because its evidence is rational. As Crdnbach et al. s< 

(1980) note, however, "If title term decision ris understood to mean 

a formal choice at a particular time between discrete 

valternatives, -decision making is rarely to be observed" (p. 84) . 

Patton (1978) reviews three competing concept iond of 

organizations that clarify the process of decision-making in the w 

real world: the rational goal attainment model; the systems 

model; and the open systems perspective (pp. 122-127) . Bedausle 

the 4 open system^ perspective most fits the context^of educational 

4 « ■ * ' * 

decision-making, decision-makers in these settings can be viewed 
as engaged in "satisf icing, " i.e., the process "of finding 

course of action that is "good enough"" (Simon, 1957, p. 204). As 

• * 
Patton (1978) points'out, the ^decision-maker in such a setting can 

never. have^ all the information needed to meet the conditions for 

rational action and even if he Or she could, it would be beyond an 

individual's capability to comprehend it all (p. 125) . 

"Satisf icing" is t)ien tenable because administrators feel they can 

change Jnost courses p£ v action if decisions ■ later prove to be 

seriously mistaken. 

Some administrators perceive evaluation as a two-edged 

sword. The pretense of objectivity embues evaluation with 

• 

* » * 

5 ' * • 
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credibility. _The administrator, who is confronted with an 
unfavorable result knows th$t any result can be attacked on 
several grounds. On the other hand, the administrator who is 
confronted with a favorable 'result may perceive the result as a 
weapon against adversaries. Firm stances, h&wever, may' not be 
ta"ken until the results dictate them. From the perspective of 

satisficipg, evaluative information cari thus be' useful in a 

»• . 

variety of ways not necessarily predictable from a "rational" 



1 

view. 



Patton (1978), summarizing the results of an empirical use 
study, writes, ^ . 

The kind of impact we found, then, was that evaluation * 
• ta research provided some additional information that was 
^ judged and used in the context of other available . 
information to help reduce the unknowns in the making 
- of difficult decisions (p. 30)\ 

'Crohbach et al. (1980) distinguish between the content of- 

compared, s i >e, "of concentrated power and responsibility, 1 * apd the 

conttex^ of pluralistic accommodation (p. 83). These contexts are 

usef^L^in describing the likely effects of evaluation results. 

^.In a context of accommodation, the eva^uator cannot 
expect' a "go/no go" decision to turn on his assessment 
of- outcomes, whereas information on outcomes is 
influential in a context of cominand — a decision all but 
made can still be unmade (p. 15^). % / * 

* - ■ * 

To ignore the process and context of decLsion-makijig in the r%il 

V . ( 
world is to be unable to adequacy examine the use of evaluation 

Information. 

i 

" Rational " non-use. More than likely, as Caplan (1980) 
writes, "There is real danger in uncritically accepting 
utilization as desirable or in being oversold on its value. Not 
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all utilization is good and not all nonutilization is bad" (p.! 5). 

t 

To label a decision-maker "irrational" or "non-r ational" becaup<#, 
. for whatever reason, he or she does not use the results of a 

t i 

pertinent evaluation is simply inappropriate. In other words, ; the 
n^n-use as well as the use .of evaluative information can be highly 
t . ational" in a specific setting and for a number of reasons. 

On© reason practitioners may "rationally?, .feel that 
evaluation studies do not warrant use is that, decision-makers may 
see the world from a different perspective than the evalaator, a* 
perspective" that may imply a different form of rationality, 
Caplan's "Two Communities" theory that addresses the contrasting 
perspectives of researchers and policy makers (see, for example, 
Caplan .et al. , 1975) suggests that an evaluator's concern for 
technically correct procedures may conflict with a program 
staff's^ implementation plan or' wijth a local decision-maker 's 

"A * 

information needs. While the evaluator sees a local program as a 

pi£ce of a educational experiment, administrator may instead 

see it as prx>vidin'j essential services to students. Wilensky and 

'Leb^aux (1958) observe' that "what the social scientist thinks of 

as 'objectiye investigation' the practitioner often takes as 

''hostile attack'" (p" # 20). Such 4 differences in the ev^luator's 

and, the pract itioner's perspectives may reduce trust and impede 

effective communication. As Deal and Rallis (1980) explail^, 

Theoretically, collaborative relationships require a 
shared perspective, high trust and power g|rity. The 
shared perspective between Knowledge .producer and 
user, however, is often characterized by <34£ferent 
perspectives, low trust and an asymmetrical ► 
distribution of power (p. 216). 



Empirical research makes clear how this situation might ocpur in 

public school settings where 

* • 

Not only have 42% of them (LEA evaluation unit heads) . 
not taught, but 70% have not run a school. This means ' 
that even when evaluation heads have teaching 
backgrounds, they, do not take the typical advancement 
route to the central office (Lyon et al. , 1978, p. * t 
66). • 

Practitioners may also f^el that evaluation results 
contradict their intuitions regarding program impacts. As Guba 
(1969) notes, "For decades the evidence produced by the 
application of 'conventional evaluation^ procedures has 
contradicted the experiential evidence of the practitioner. 
Innovations have persisted in education not because of the ^ 
supporting evidence of evaluation but; despite it" (p. 1). In 
David's study of the local uses of Title-I evaluations (1981), 
for example, practitioner^ expressed strong feelings that, 
regardless of • what the data .showed, their program had succeeded 
(p. 38)^ Such situations frequently occur when summative 
evaluations find "no significant differences" associated with a 
program in the face of the practitioner's direct experience that 
the program really did substantially alter cla&sroom life. 
Shapiro (1973) argues, "While it is important Jpo try to explain 
negative (i.e. 1 statistically non-significant summative) test 
results, it is far more important to account for_J:he 4 disparity 
between the negative test "findings and the clear differences 
observed in classroom behavior" (p. 527) . The Joint Committee's 
Standards A3, "Infornfetion Scope and. Selection" (pp. 1 27-31), and 

Q10, "Justified Conclusions" (pp. 135-137), point to the 

/ ■ 
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importance of sugh accounting in .good evaluation practice. 

Another "reason for the rational non-use of evaluation 

findings -relates to the environment of local school settings*. 

Non-use constitutes a rational choice when institutional, 

political, or the administrator/ s own goals that ,ar^ quite 

iistinct from formal program goals suggest it as a viable course 

of action. For example, Granville (1977) explains that "a 

decision make), in addition to .considering whether or not a 

program has fulfilled its manifest objectives, must also consider 

its fulfillment of latent 4 objectives, such as enhancing the 

agency's prestige or expanding its resources* (p. 2). \ This, ^ 
* 

theoretically, may not.be an ideal basis for decision-making, but 
in practice administrators clearly have their own ■ survival needs 
and personal agendas that cfan affect their use of evaluative 

information. The desire for personal prestige^, power--or merely 

%■ 

for keeping /One's* job — may make nrdn-pse imperative. 

£ * ^ 

A related reason why results may rationally go unused is 

that some evaluation studies were never meant to be used. As 

Alkin (1976) explains, this is frequently the case when an 

.externally mandated evaluate iok is involved: 

Many practicing school administrators . . . believe 
that evaluation is simply an event that leads to- 
compliance'with various agency requirements. There is 
no real expectation that major basic decisions will be 
made. - The name of the game is simply not to get 
"dinged" by the governmental agency, (p. 16) . 

In v a recent article, Alkin (1980a) made the same point using the 

analogy of a garden party. 

... suppose the fcost . . . should insist that ,each 
of the guests periodically rate the quality of the 
party, or the*, drinks or the food, etc.-^it can't 



sW hM, a f^ art - Qf the " price of admission," so to 

^div k idua u Lipr°S) : realiy chang *, the behavi ° r ° f ' 

Indeed, deliberate non-use is thoroughly rational if an 
-equability assessment suggests that a program 'is not ready, for 

evaluation (see, for. example, Rutman, 1980), but, because it has 
been mandated, an evaluation must be conducted anyway. 

A final reason for non-use is that some evaluation studies, 
.are poorly done or technically weak and do not merit use. Cuba 
and^tufflebeam (1970) observe: 

' " evSuaHon^!?^ 8 ™ ke " ron * assumptions' about what an 
blsed £ ^hlii *l Sh ° Uld a . CC0 »P"sh, and . . . (then) 
fSHt bad ^L! rr ° neOUS assumptions, ?esearchers 
wactitifnlS 1Ce a upon ^suspecting and unsophisticated 
usually uspif; c ? S J ? onse ? u ence, evaluations are 
iusH fL* • ft' and Petitioners are largely 
taken ita„J .S? 3«?ndiced view they, typically have 
taken about evaluation and its utility (p. 6)\ 

Ironically, empirical evidence exists CAlkin et al. , 1974, p. 48) 
linking .poor quality evaluations with poor quality programs, the 
very projects that/ 'theoretically, good evaluations should be i 
able to help md^t. As the Joint Committee's Standards .( 1981) 
would suggest, methodological quality is a minimal requirement 
for. good evaluation practice, without tL,-the use" of results 
may be inappropriate. 

• Evaluators are aware that poor technical" quality can occur 
in many forms.^ An all too common situation that justifies 
non-use occurs when evaluators do not attend to a program's 
actual degree of implementation. As Guttentag and Struening 
(1975) observe, "Obvious though it may seem, evaluations continue 
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without either raising or answering the primary question*: "Does 
the program (even) exist?^'(p. 4), This, situatioft continues 
despite t^hfe availability of several strategies fq? -toeasur ing 
implementation prior t<J comparison of results for program • 
participants an<^ non-participants (Revicki &• Rubin, 1980) , 
inclu/ding notably the model developed .by Hall ? and Loucks ^1977). 
Standard Dl , '"Object Identification," encourages evaluators to 
examine the object of the evaluation sufficiently so that its 
"form(sJ • • can be cleanly identified" (Joint Committee, 1981 , 
P. '99). 

Taken together, then, these reasons for the non-use of' 
/evaluation results suggest first^that "rationality" for a local 
decision-maker may differ from mote traditional definitions of 
rationality. Evaluators must recognize that administrators at 
times may^not behave rat ionally\ in the classical sense."' ' „ 

Administrators' decisions may^ be rational only whe~n viewed from 
within their political ■ frame of reference. 'To assume. that 
administrators' wirf apply the. results of , evaluation in 
predictable ways is then unrealistic. Local usersymay see ; 
evaluation as external and irrelevant to the program; the 
immediacy of other concerns in 'the local context, the potential 
mistrust of evaluators, and thedif f iculty of^Vccept ing criticism 
graciously may cause local decision-makers to use or not us$ 
evaluation information in their satisficing. Also, as Weiss 4 
(1979) explains, because evaluations can end in unclear, 
incomplete, or 'conflicting results/ the "rational" choice for a 
decision-maker is frequently not evident, and to label an 

ii 



administrator nojpkional in such cases makes little sense (pp;. 

These reasons also suggest that,, in an important sense; 
non-use represents a viable form "'of- "use." The majo'r- task for 
evaluators is to maximize the appropriate use of evaluation 
^resul^s. If the non-use of results is appropriate*, then non-use 
is highly rational and .should not be condemned. The implicit 
t assumption that use is good-and norj-use bad {We'iss,. * 1979, p. 7) 
i?ust, like the myth of rational evaluation use, be recognized as 
untlrue. . Weiss ^1980, cited in Kennedy, Apling, and Neumann, ' 

• 1980) identifies, five characteristics of rational decisions: (1) 
purposiveness; (2) bcundedness of time, actors,, and events; (3) 

• calculation, i.e., "'consideration of the aptness of various 
actions for dealing with the problem situation and . achieving 
results";^) perceived significance; and (5) realization -that a 
decision is being made (pp. 132-133). Given, these 
characteristics, the non-use of evaluation results in educational 
settings may frequently ' represent ^a rational decision/ 

The Concep± of Use 
'Anyone familiar with local evaluation settings knows that 

« 

the uses to which Evaluation results can be put are as varied as 
the types of information that can be labeled ".evaluation. " As 

« ■ 

mentioned above, in some -cases ^evaluations are conducted only in 
response to federal mandate, and the non-use of results occurs 
because the evaluation is an externalized event standing apart 1 
from the immediate concerns of program staff. In others, a 



decision-maker '^ea^er to impress someone may wafit to'be the first , , ^ 
among his or her pe.ers to commission an evaluation, and the 



results are almost beside the poijifc. In stirii others, a series 



almost beside th 

of eyaluation v re&ults over a* period of time may alter or confirm 



a user's ideas about a specific program or about a general 
approach to\a problem. The list goes on. Any discussion of th$ 
Use of evaluation.. information, then, must first attempt to define 
the key term fise so that instances of use can be separated from \ 

* * ' v - * . ' ;. J \ 

those of non-use. Only in this way can we ,hope to conduct 

i 

research to determine factors affecting use t in order to enhance 
the effect of evaluation results. As Weiss (1979) 6as written, 
" Until we resolve questions about the ^ def inition of use , we face 

^a ^rufrure of non - comparable studies of use apd scant hope of 
cumulative understanding of how evaluation and decision s- making 

- intersect " (p. 13, emphasis in original). 

The boncept of evaluation use has changed dramatically over 

the past eight years'., There is what Alkin et al.' (1979) call tfie 

i * • 

"mainstream" viewpoint, that 

'I . . 'contends that evaluations seldom influence 
- program decision-makers and holdfe out little hope that 1 
evaluation will ever breakthrough* the barriers to-', 
real impact on programs (plr 17) . * • < 

Implicit in the gloomy portrait of evaluation's failure is an 

unrealistic, rationalistic point of view that assumes the results 

of evaluations should ^fceate immediate and observable effects. 

The "alternative" viewpoint — that "evaluations do Already 

influence programs in important and useful ways" (p. 17) — has 

been induced from the combined results of erhpirical use studies. 
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"Taken together, these results suggfest that a possible reaaor^ for 
— * 

the*generally""i^egative. lmpress<£en of evaluation use conies frdirr^n' 
inappropriate definition of what counts as^ use. * Although the 
focus of these studies is generally on federally funded programs, 
- there are ^implications ^*f or the discussion Of evaluation use at 
the local le^el in general. Consider the following studies. 

— Alkin, Kosecoff, Fitz-Gibbon, & Seligman (1974) examined • 
42 ESEA Title vfl evaluation studies, conducted in 17 states 
during 1970-1971, measuring impact at both the federal ard, local 
levels. Almost every project director— 39 of the 42— reported 
that the formative studies were useful to them (p. 26) , and "no 
prefect director indicated that he would prefer not to- have an 
evaluator on the" project" ' (p. 28, emphasis in original). 

.# 

— Pattoji et al. {1977) interviewed decision-makers and 
evaluator^ from 20- federal health evaluation studies. ^Their 
results indicated that evaluation findings -were used ^3? 1 
decision-makers, 3 ?but not in tfte cl.ear-cut. and 
organization-shaking ways that social scientists- sometimes - 
believe research should, be used" (pi 144-) . Instead, evaluation, 
results provided another piece of a complicated information - 
puzzle that helped to r'educe-'the uncertainty wi£ljin which federal, 

decision-makers operated.- • - . . 

• ) • A 

—Alkin, DaiIlaTrrir white ^979) did intensive* dase studies 

of five ESEA^tle iSsr Title IVC programs", -locating inly one 

instance of mainstream use, i.e., a' case where a formal 

comparative evaluation was. conducted", "an evaluat^n report was, 

written, the findings were- negative, and the. program was dropped" 
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t (p. ^224). What they did find in each case, however ,^&s an 

• indirect kind of use. Decision-makers stated that the evaluation 

* * r 

results influenced them in gradual or general ways. 

—Dickey (1979b) , studying 47 ESEA Title IVC projects in 
Minnesota, found- a high utilization level; 72% of th§ 

i ^ 

decision-makers reported the evaluation results to be *very" or 
"quite" useful. 

j 

— Kennedy, Apling, and^Neumann (1980) r foc&sing on local 
uses, .discuss the results of 345 interviews with LEA personnel in 
18 districts known for successfully using evaluation and testing 

information. They describe a variety of use issues in four 

* -» 
clusters of applications:' districjt-wide, program-level, 

building-specific, and qlinicaX. * 

Brown andu Br a^kramp . (1980) summarize the current view of 

evaluation utilization: ". . . effective utilization do^s not 



upn utilization: ". . . effective utilization do^s not ^jf 

necessarily mean that any of the recommendations are implemented 

or that there are any immediately apparent decisions based on the 

information" (p. 91)/ This suggests- both that the discouraging* 

picture of evaluation use'created by the quotations tibove may 

stem from a mistaken expectation regarding the nature of use and 

that "the extent of evaluation yse may therefore be underestimated 

in the literature. Wise (1978) has suggested that • 

If there is an evaluation utilization problem,, it is u 
not that decision-makers do not use the information* 
they recei#£, i\t is that evaluators carwiot easily see * 
. their information be img used iri the incremerttalism of 
real-world decisionmaking" (p. 24). 

Similarly, Alkin, Daillak, & White (1979) argue that 

Takeri together, the studies and our observations and 
experiences "suggest to* -us that evaluation can make a 
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> - * 

difference, that it does so .more often than the 

published *cr itiques suggest, that some school 

districts characteristically produce a high 

proportion o£ useful evaluations, ahcf that some 

evaluator s Jiave acquixed^skills. Jbhat allow_.them._to _ _ _ _„ _ 

carry out technically competent and programmatic ally 

influential evaluations (p. 16, emphas is in 

original) . 

Those instances where evaluation results are directly translated 
* 

into a decision are sufficient, but not necessary cases of use, 

* « 

Based on their empirical work, Alkin, Daillak, and White 

(1979) define an "instance of the utilization of "local school . 

program evaluation" as 

Evaluation information considered by a local client, 
sanctioned local users, or external users as a dom- 
inant influence, one of multiple influences, or one of 
multiple, cumulative influences in making decisions, 
substantiating previous decisions or actions, or 
establishing attitudes related to establishment, ex- 
ternal funding, local district funding, continuance of 
a qomponent, curriculum/instructional methods, 
administrative/personnel operations, or community 
* acceptance of, the local school program (p. 232) . 

This definition reflects the "alternative* notion of evaluation 

use discussed above, recognizing, thai!' the evaluation information 

may be only one 'source of information used in a complex setting. 

fc Considerably more "direct, Smith's (1980c) definition of 

evaluation use~-the "conscious employment of an evaluation (its 

processes, products, or results) to achieve some desir^£^f!dPor 

impact" (p. 25)— reflects the purposeful na,ttfre of use implicit 

in Alkin et ai:'s definition. By including the process, product, 

and results of evaluation in his definition, Smith further 

suggests the nee<3 to distinguish the effects of the evaluation 

process* from the effects of * its outcomes. As Cronbach e^ al. 

(1980) note, , 
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• • . whether an evaluation i$ launched to promote a x 
cause r or to report -neutrally on events, the measurement 
procedures and reports can, easily have a wholly 
% unanticipated ^nfluence^o^/What happens next (p. 27). 

Knowing that the process itself can result In changes (e.g., 

people, knowing they are fc/eing evaluated, working hard to look 

good or perhaps to discredit the evaluation) , to study use we must 

trace the evaluation's effects from its inception until well after 

its completion. This definition appropr iai^ly recognizes the 

dynamic nature of use. 

Smith also distinguishes use from impact, • which he defines 
as "any discernable 'actions, events, or changes in conditions that 
are directly influenced by the evaluation, its processes, 
products, or f indings n , (p. 25). « Given this definition, an 
evaluation needn't be used to have impact. If, for example, 
teachers Work to oust the superintendent who introduced an . 
accountability system', the evaluation has had an impact, 
regardless of its outcomes^ or % the use to which the superintendent 
puts it. This distinction is helpful ih examining the varied 
effects any evaluation can have. 

Leviton and Hughes (1979) present two, necessary conditions 
for uset (1) Cook and Pollard's (1977) suggestion- that there be 
"'serious discussion of the results in debates about a particular 
policy or program" 11 (p. 4) j and "(2) " . . . evidence that those 
engaged in policy or program activities would Save thought or 
acted differen.try in the absence of the research information" (p. 
5) . If the notion of "serious discussion* is changed to "serious 
consideration," following klkin, Daillak, and White (1979), the 
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.first condition is both a necessary and sufficient condition of 
evaluation use. When a decision-maker actively and consciously 
considers, h&w he or she yill use^fn evaluation — even if this is 
simply an awareness of how an evaluation may frighten certain 
people — the example is one of use. Eliminate this conscious 
element, and^the example becomes one of impact. 

In contrast, the second condition can be considered ' neither 
necessary nor sufficient. for use at the local level. It was 
suggested above that non-use^may in some cases be an important 
form o'f use.. Decision-makers may not think or act differently as 
a result of seriously considering the results of ah ^evaluation ; 
they may react to the findings and choose to ignore them. To the 
extent that information once received changes a person's thoughts 
forever, the, second condition may be 'technically "necessary," but* 
because observers may be hard pressed to find evidence of this,^it 
seems pointless to make such a case. Further, the second 
condition alone is not sufficient to determine an example of use 
because, as stated above, without the conscious consideration 
marked by the first condition, the example Is one of impact rather 
than use. Merely seeing activities reflecting change is not 
sufficient evidence of evaluation use. 

Figure 1 presents these- id6as in' diagram form, h 
decision-maker's or -information user's intentions will result in 
the use of evaluation' in* a variety of wayg. These are 
simultaneously examples of evaluation use ^ana impact. All other 
outcomes, i.e. those unintended by the user, are examples of 
evaluation impact. Both categories represent impact in that the 

18 

31' 



evaluation directly influences "actions, conditions, or changes in 
conditions." Gi^en this conceptualization, the notion of indirect 
influences is no lori|er helpful; either a user consciously uses 
results (an example of use) or, without a user's intention, 
results create a direct effect (an example o^impacb) . 

Consider, for example, an evaluati^ri^hat/ indicates serious 

. / 

weaknesses in an ESL program for Hispanic children. Once the 
results are available in any form, use could include a central 
administrator's decision to remove the project directory a group 
of parents' complaint to the school board that tJfeir children are 
being poorly served, or a teacher's decision tp fight for the 
program due bj^ the evaluation's inadequate ir^asurement procedures. 
For each user, the evaluation may have additional, unintended ' 
impacts. The central administrator, for example,' having fire<^the 
project director, may be dismayed to learn of £he parents' 
complaint because the director may have been an effective 
community liaison. Erom this ^xample, it i,s clear that one 
person's use can be another's impact. 

Several writers have outlined conceptual dimensions of 
evaluation use (Alkin, 1975? Patton et al., 1977;' Alkin, Daillak, 

and White, 1979; Conner, 1979? Weiss7 y "l979; Brown, 1981). Weiss 

* ■ * 

(1979) has suggested six dimensiohs* for deciding what constitutes 
t. 

--a use that would provide qonceptual clarity for studies: 
1* What is used? 

2. How, direct is derivation from the study? 

3. ' By whom is it used? * 

4. By how many people? 

5. How immediate is the use? 

6. How much effect is required? P (pp. 11-12). 
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Brown's (1981) conceptualization includes five dimensions: (1) 
Who uses the information? (2) For what purpose is<ithe information 
used? (3) What is the evaluation context? (4 1 ) To what extent is 6 
the information used? and (5) What kinds of information -are. used? 
(v. 3). s . • ' 

Presenting^ categories derived empirically, Alkin, Daillak, 
and White (1979) provide eight categories in their analytic 
framework of local evaluation use: preexisting evaluation bounds; 
orientation of the users; evaluator's approach; evalu^tor 
credibility; organizational factors; extraorganizat ional factors; 
* information content and reporting"; and administrator style (p. 
235) r , 

Several conceptualizations of types of use have^also b f een ; 
offered. For example, Fullan (1979) has 'suggested that 
information may result in changes in values, in understanding^in 
roles, in organization, or in materials. Some authors use 
different terms for types of use that apparently involve the same 
processes. As Weiner, Rubin, and Sachse ^(1977) observe, "These 
# categories 'are neither mutually exclusive nor exhaustive" (p. 
12). - . L 

Instrumental ,use , also called allocatlve use (Weiner, 
Rubin, and Sachse, 1977) , represents the traditional, mainstream 
view of the use of evaluative information .where the information^ 
* resfilts directly in programmatic changes. As noted earlier, this 
type of use rarely occurs, aithough some examples of instrumental 
use can certainly, be identified (e.g.., the Rockland case study in 
v^Alkin, .Daillak/ and Wfyite, 1979). Pelz (1978) reminds us that 



21 



9 

ERIC 



34 



V • • it is seldom possible to trace a single decision^ to a 

single piece of knowledge in a one-to-one matching between input 

and output*" (p. 354), and Weiss (1979) ^concurs: 

* To limit our attention to direct and immediate 
application of evaluation results to decisions' 
forecloses the opportunity to understand how 
evaluations in fact affect program operation (p. 7). 

A second, more common form of use is labeled conceptual use , 
i.e. use "influencing a policymaker "s thinking about an is^ue 
without putting information to any specific documentable use" 
(Ri.ch, 1977, p. 200; see also Caplan et al. , 1975). Rein and 
White (1975) suggest that over time, summative evaluations change 
people *s ideas of what can or can't be accomplished # in programs. 
The cumulative effect of ^conceptual use and a variety of other 
information then determines future programs. The importance of 
this type of use is noted by Cronbach et al. (1980) who* wjite that 
"stimulating a discussion that leads *to gradual phange in 
prevailing views is very likely the most* important effect of 
evaluation research" (p. 193) . They even go so far as to say that 
"only a minority of evaluations are to % be judged primarily by 
their service to whoever commissioned them" (p. 61) , adding later 
that an evaluative study Ms to be judged primarily by its 
contribution to public* thinking and to the quality of service 
provided subsequent to the" evalyattwi" (p. 64) . This type of use 
lias also been labeled appreciative* (Weiner, Rubin, _a#d Sachse, 
1977) and ".enlightenment**" (Cronbach et al. , 1980) . . * 

The distinction between instrumental and conceptual use may 
„be clearer in theory than in practice. Leviton- and Hughes (1979) 




have written that "it is difficult to determine where conceptual 
use ends and instrumental use begins" (*p. 10) , pointing to the 
need for documentation of ^instrumental use; an example of 
instrumental use may be mislabeled because of missing^ * i 

documentation. Weiss (1979) notes that the use of rejsearch and 

v 

evaluation results can more sensibly be placed on a continuum, 
with actual, immediate (i.e. instrumental) use at one end and the" 
conceptual, diffuse contribution to understanding at the other (p. 
10) . For example, any given evaluation will have certain effects 
on the program it studies — textbooks may be changed, objectives 
rewritten, services expanded, and so on. At the same time, the 
results of that study contribute to an overall picture of 
education' in the decisifcn-maker "s mind, one of many bits of such 
information integrated yearly. 

A third type of use has been variously called symbolic 
(Pelz, 1978? Young and Comptois, 1979) , persuasive (Leviton and 
Hughes, 1979) , or ritualistic (Braskamp^ 1980) use . Although 
these labels point to slightly different types of use, the 
distinguishing characteristic of symbolic use as an umbrella term 
is the pre-eminent influence of the evaluation's context on the 

use; t^he success of such use is independent of the message brought 

\ t 

l^o the decision-makes (Weiner, Rubin, and SachSe, 1977, p. 13*). 
Symbolic use car> take several forms. For example, program 
personnel who solicit evaluation^ only to satisfy external funding 
agencies are engaging in symbolic use; the evaluatiorf is only used 
to persua4e the agency that the game is being played according to 
the rules. In persuasive use, decision-makers "[draw] on 
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evaluation evidence in attempts to convince others to, support a . 

political position, or to defend such a position from attack" 

(Leviton and Hughes, 1979, p. 7). Knorr (1977) indicates another' 

symbolic use of evaluations; she writes that Austrian 

administrators reported they had used evaluations to legitimize 

decisions (pp. 171-172) . Further examples of symbolic use include 

gaining recognition foj: a successful program or discrediting a 

disliked program (Weiss, 1977); and as window dressing or as part 

of a public relations campaign (fiJLkin, 1975) . 

To label symbolic use ritualistic is a misnomer since the use can 

have deliberate and important program impacts. Cronbach et al. (1980) 

point out that "sometiraea an evaluation or monitoring procedure has 

only a ritual or symbolic purpose. To use these labels scornfully is 

to misperceive the utility .of symbolic acts" 4pp. 159-160) . SuOh scorn 

also ignores the political realities of most evaluation settings. 

Figure 2 diagrams the relation 6f these modes of uses to the evaluation 

context. In cases of instrumental and conceptual use, evaluation 

results are .applied to the political context in immediate -or longterm 

ways. In contrast, symbolic use comes from the political context; 

knowing what they want to do, decisionmakers turn evaluation results 

to their own ends, whether appropriate or not. To do this is not 

necessarily unethical or manipulative. It may be a method for 

survival. Patton/ ( 1978) has written, 

The potential for enhancing utilization lies less in its 
capability for rationalizing decisionmaking than iji its 
% capacity to empower the users of evaluation information (p. 
35) . 

The symbolic use of evaluation recognizes this. 
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Figure 2_. Types of use and related actions 
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The conceptualizations of use presented above may accurately 
rcflecc real, world evaluation dynamics, but the shift from the mo r* 
traditional view of use does produce some problems. An expanded view 
of use makes it more difficult to study use phenomena. For example, 
"Tt is literally impossible "to prove'- (conceptual) use" (Pullan, • 
1^80, p. 44) , and few decision-makers may be wilLmg to recognize or 
own up to symbolic use. Nevertheless, the trade-off of insurability 
in return for a more realistic perspective on use is clearly 

worthwhile. As Braskamp and Brown (1980) have arg,ued', "although 

I 

the expanded definition makes utilization less dramatic \^nfi more 
difficult to explicitly measure and demonstrate, it represents a vievf 
> of evaluation in which the role of human interaction in the 
communication process is given more credance" (p, viii). 

Empirical' Studies of Local Evaluation Use ' 
Dickey (1981) expresses the consensus in the literature when she 

1 

writes, "There have been many articles on under-ut ilizat ioni, but few 4 

>+ * , i 

empirical studies suggesting what variables might account f|>r it. 
Supporting data are limited, and most of our 'theories' com| more from 
common sense than from research" (p. 65) . In 1975, Davis aAd Salasirv 
reported that a "review of 1,200 references on [knowledge] uti ilizat ion 
contained only 2 1/2% which pertained to evaluation.-. . . dven in the 
broadest sense? (p. 626) . Conditions were similar in 1979 when Alkin 
and Daillak wrote that * * 

While much has* been said and written about \he* problems 
begetting evaluation and about the. underut ilization of 
evaluation information, very few empirical studies of 
evaluation -utilization have been conducted. Most of the 
literature is anecdotal in form' (p. 41) . 
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Shapiro (.1979) agreed: "The literature on utilization, both applied 
and' theoretical, tends to be ^ad hoc and nonr igorous" (p. 1). Cook 

to 

(1978) has suggested that "the quality and imaginativeness of most 

/ 

(but not all) utilization studies leaves something to ke iesited" (p. 
14) . * / , . _ • . • 

The difficulties of conducting research on evaluation use are 
many* Most researchers would agree that # v 



e researcher who truly wishes to understand the "why?" o"f 
utilization cannot treat evaluation as a bl*ck box^with r i6 
A . inputs (characteristics, factors, etc.) and outputs ,; 
(decisions) , but must open up the evaluation black box and* 
carefully study the interactions of people and -events which 
produce the multiple consequences of evaluation and which 
give these consequences meaning (Alkin, Daillak, and White, 
p. 32) . 

But, even apart from the conceptual issues discussed above, practical 
constraint^ on/studying the use of evaluation information abound.^ 
Leviton and Hughes (1979) note that it is often difficult to document 
use or to demonstrate tEat change at any level is due primarily to the 
results of an evaluation (p. -15). Decision-makers and their local 

> - 

school personnel may be reluctant to share their use — or non-use — of 
evaluation results* Stevenson (1979) writes that. "verbal acceptance 
of .findings may not be followed by appropriate action. Verbal 
rejection of findings may be^ followed by actions which imply 
acceptance" (p. 3). A further problem stems from the^longterm effect 
of results; a study completed one year may continue to affect programs, 
years after i-ts final report is released (Holley, 1980b, p. 106) . 

Weiss (1979) giv.es five cases in which the interaction of 
evaluation .results with a program andj3ecis ion-male^ r leave the 



evaluation researcher in a quandry: 1) a decision-maker given an 

7- 
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evaluation study without definitive answers; 2) a decisis 
two conflicting evaluation studies; 3) a declsi6n~m*Jc€r 
program with ho real problems or decisic& jtoints; 4) a incision-maker 
considering many' factors, gnly one of which is the evaluation 




information; 5) a decision-maker .using, evaluation results symbolically 
to justify what was pre-plannej3^pp. 3-6) . The question »f what 



appropriate "use" looks .like in these cases is& a difficult one to 
answer. 

Despite these difficulties, a small body of, empirical research 
on the local use of evaluation results does exist. Because !tru$ 
experiments on this subject are virtually impossible, the existing 
studies^have been of four types: interviews (Alkin et al, 1974; * 
Andrews, 1979; bavid, 1981;' Didkey, 1979; Kennedy, Apling, and/ 
Neumann, 1980-? Patton "et al. 1977; JWilliams and^Bank, 1980); surveys 
(Caulle? ajid Sfnith* 1978; Goldberg, 1978; Lyon et al.', 1978? king JQk x 
Thompsdh, 1981} ; ; report simulation studies (Granville, 1977; Newman, 
Brown, and Braskamp, 1980; Thompson/ 1981; Thompson and King, 1981);" 
and naturalistic case studies (Alkin, Daillak, and White, 1979). 
These studies have' addressed three major* questions: 

1. Are evaluation results us^d? 

2. What are the characteristics of, local evaluation units ^nd 

v ' * 

off local decision-makers? 

3. What factors affect ~us<f.at the local leVel? 

Tentative* answers compiled from the literature provide some insight 
into the local u£e of evaluation results. 

1.^ Given th? s$nse in the fipld that* results were not being' 
,tfsed, much e*arly research .tried first to determine ff results were 



/ 

being used and/ if so, in what ways. Some of these studies have beer 

mentioned above in discussing the changing notion of evaluation use. 

The good news is that evaluation results are used. For example, 

although they studied the use of federal health evaluations, the 

findings of Patton et al. (1977) have been widely cited as providing 

support for the use of evaluations. In ^open-ended interviews with 

three informants per project, i.e. the project officer, the 

decision-maker, and the evaluator, they found that* 1 

. . . evaluation research jjs used by decisionmaker~s v but not 
in the c^ear-cut ^and organization-shaking ways that social 
scientists sometimes believe research should be used (p. 
144). • 

Studying the local uses of Title I evaluations at 15 sites, David 

'(1981) found that the results were \ised, but primarily "to meet, legal 

requirements, to provide feedback, and to provide gross indicators of 

prpgram effectiveness" (p. 31). As she summarizes, "Title I 

evaluations do not seem to serve as primary purposes, either as a 

i^a^s on which to guide the program or as a guide to program 

improvement" (p^Jl) . The use of evaluation results and other 

information are used is implicit in Kennedy, Apling, and Net/mann 

(1980) where the results of 345 interviews and the^observations of- 

roughly two dozen meetings are organized into four categories of 

issues: district-wide, program-level, building-specific", and clihical 

Varying uses are discussed £or each issue category. 

The perception of ^ocal users also supports the claim that 

results are used. Dickey (1981) conducted a retrospective interview 

study of 47 Title IVC projects using ati^lyses of evaluation reports 

and archival data as wetl as questionnaires and some telephone 



interviews of local project directors. As mentioned above,- over 

S V 
two-thirds of these users reported finding the evaluations "very" or 

"quite useful" (p. 71), -suggesting that they perceive the results 

being used. In a nationwide survey of LEA users, King and Thompson 

(1981) similarly found that over half— 60%— reported finding the 

results of program evaluations either "ve#y useful" or "useful" (p\ 

The seminal case studies reported in^lkin, Daillak, and White 

(1979) further suggest that evaluation results are used a'ncf in a 

variety of ways (see, for example, pp. 223-226). These naturalistic 

studies marked an important methodological addition to evaluation *use 

research in that they involved on-site observation and interviews. - 

The question, "Are • evaluation results used," has been answered 

empirically, and the answer is yes. _ „ — - 

'* # 2. The second question addressed by empirical research has been 

that of describing the characteristics of the participants -in local 

evaluation use, i.e. the evaluation units and the local 

t decision-makers. Two nationwide surveys of large city evaluation 

units have been conducted, (Lyon et al., 1978; Webster and Stufflebeam, 
P . 

1978) . The portrait that emerges of these evaluation units — and it 
* x 

should be noted that little is known about evaluation in small 
districts, except that they, are unlikely to have an evaluation unit 
(Lyon et al., p., 8) — provides little to comfort the evaluation- 
community. < 

According to survey results from over* 200 large districts (Lyon ' 
et al., 1978), evaluation is an in-house actdvity. School district 

V* 

personnel, rather than external consultants, do the magor share of 
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evaluation office work, and, contrary to popular belief, these units 
are primarily supported by local funds, not federal or state monies 
(p. 57) Although little agreement exists on what constitutes* basic 
evaluation practice or on what evaluation activities deserye priority, 
<p. 90) , evaluation in these units fjteqtjently means achievement 
testing; roughly 75% say that "student achievement is the dominant 
topic of data collection", (p. 76) j and an eqt^al percentage say -that 
"testing is their major method of data collections" (p. 79) . The two 
activities^ ranked as consuming the most time*--the assessment of 
instructional programs and of student achid^^erft of objectives-- 
.suggest a narrpw view'of evaluation (p. 83).- Furthermore* data 
collection typically repeives more attention than^ata analysis (p. 
77) , and only 15% of responding offices reported that the use of 
evaluation -results to modify 'program^was either first, second, or 
third among time-consuming activities (p. 86);, x 

The relation of evaluation to improved isj struct ion is then 

, -\ 

tenuous, especially given the organizational position of many units/ 
Evaluation units are more likely to be* in one of 'the typical lines of 
authority (e.g. Instruction or Administration) than to report directly 
to the superintendent, but, even soJN>2% of the officios are not 
located in the Instructional line. TDeveippment acfT>ities in these 
offices generally center on tests* and evaluation instrument^, rather 
than on instructional programs and prcjducts. This partly explained 
hy 'looking at the clientele of tehe a^rage upit; roughly 60% of the 
time is spent with administrators compared to only 40% with 
instructional clients (p. 102) . Also, the career patterns of many 
evaluators may de-emphasize Instructional management experiences: a 



- ^ 



full 42% of evaluators responding have not taught (p. 66); 

^ Th,e results of Webster and Stuff lebeam's survey of 35 large 
urban school districts (1978) provide additional details of this 
portrait. If dollars invested suggest commitment to an activity, 
e/aluation functions are not strongly supported in these districts; 
the "total evaluation bill [for 1977-1978] was $13 , 002,049, a mere 
fifteen-hundredth^ of one percent of the je^uoation budget" (p. .26) • 
As the Lyon et" al. study found, the evaluation departments reported 
putting "most efforts into testing, product evaluation, and data 
processing" (p. v 29) , rather than into program improvement^ Because 
all evaluators are influenced by the politics surrdunding them' (p. 
34), evaluation department heads reported having to "[sell] evaluation 
activities to decision-makers, in many instances, convincing them that 
they need information to make better decisions," (p. 29) . Even in 



districts with evaluation units, then, the results of these surveys 
sugges^U&t^ tlie use of evaluation results at the local level' is a 
less* than likely prospect. 

In^addition to. these surveys, Kennedy, Apling, and Neumann's 



* (1980) interviewsof evaluators in IB districts across the country 
provide further insight into the Xocal evaluation office to suppoft 
this view. The evaluators interviewed discussed five issues, ^irst, 
the multiple^clients they serve in a local setting make it impossible 
to focuS on or please everyone all of the time. Second, the conflict 
between object iv % ty ..and involvement forces evaluators to create a 
delicate balance between independence and close working relationships 
with ^ro^is^^^naiqB^ Third, ^the concern for professional 
credibility means they must aim for technical quality in difficult 
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working environments, where* cl ients frequently question the 
educational (but not the technical) quality of. trfeir work. Fourth, to 



promote information us^, evaluators need to employ several strategies 
, such as engaging in early negotiations with potential users and 

securing the support of board policies that mandate evaluation use. 
.Fifth, the relationship between evaluator cfnd client was reported to 
influence use only whfen the evaluator became a personal consultant, an 
extremely time-consuming £ask, thus suggesting that it "may not be 
reasonable ... to place too much responsibility upon the evaluator 
for improving evaluation use" (p. 118) . 

The characteristics of local decision-makers derived from 
empirical studies provide additional information ^pertaining to local 

use. * As noted above,, these users frequerftly report finding report 

"* » • • 

results useful. Alkin; Kosecoff, Fitz-Gibbon, and Seligmaft (1974) 

studied the relation of ' evaluation -^reports ^ap4 decision-makers in 42 

Title VII projects. ~* The" methodology consisted *o£ data collection 

through the analyses of ^valuation reports and audits, retrospective 

questionnaires or telephone .interviews with project personnel, and 

federal monitors* ratings, followed by extensive statistical analysis. 

In computing a local decision-making factor, they found that "when 

[local] project directors reported the evaluation useful in one area 

(e.g.* in preparing reports, changing personnel, "recommending project 

Changes, etc.), they generally reported it utefut in all areas" (p. 

53). Furthermore, / ® 

/ m Whereas evidence of evaluation sophistication and compre- 
hensiveness . . . were associated"with federal decision 
making, local decision makers were influenced by the 
physical amount of- evaluation irfformati&n produced and 
genera> judgments of project quality, ftith respect to 
local- decision making, the rule seemed to be? the more 
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• C evaluation the better and the more positive the 

assessment the % better (p. 65, emphasis in the original)* 

The image of the local evaluation user derived from this study is 

someone eager for information. It should be noted, however, that the 

study also found no^^corded failures in the evaluation reports (p. 

17); the possibility that local decision-makers were finding useful 

positive documentation of their programs might easily explain why the 

evaluations were appealing. 

# 

In the study cited earlier, Dickey (1981) found that 

. . . the project director is most likely to use the 
evaluation when he or she is interested in and committed 
to it, when he or she sees its procedures as appropriate 
and its recommendations as helpful, and when the* final 
report is produced on time for the project's needs (p. 
,73). 

She also found t^at directors of validated projects were more likely 
to label evaluation .information useful. This conflicts with 
Granville's finding (1977) that, in a simulation study, principals 
responding to a report on an innovative program were more likely to 
incorporate objective evidence into their decisions than those reading 
a report on a "routine" program (p. 6). These results may clearly be 
due to the "numerous methodological differences between the studies, 
but the variable of progWia^nnovatiyeriess needs, additional 
clarification. 

The four types of issues raised by locaL users interviewed by 
Kennedy, Apling, and Neumann (1980) are presented^ Table 1 and 
suggest the variety of concerns these multiple clients face' .in using 
evaluation' and testing information. The problems and information 4 
needs of local users' therefore differ radically. Smith (1980b) found 
that school board members had a different perspective toward 
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U Issues raised by users in Kennedy, Apling, and Neumann (1380) 



, r- 

Issue Category 


Topics of Concern / 


District-wide 


' r J 

a. Federal influences 
bf\ Changing enrollments 

c. Student behavior 

d. Testing 


Program level 


a. Responsibility and authority 

b. Compliance 

c. ' Funding 


Building-specific 


a. Implementing dis.trict policies 

b. Building management 

c. Program improvement 


Clinical 

■ .. 


a. Students 1 instructional needs 

b. Students 1 social and emotional needs 

c. Referyal of students to special programs 
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accreditation studies than administrators; the users who provided the 
• . lengthy set of issues in Table 1 similarly must view evaluation from 
differing perspectives. * 

Three studies, have examined administrators' and project staff's 
attitudes toward evaluation and evaluators. David (1981) discovered 
two underlying attitudes toward evaluation among the Title I local 
staff: a "narrow and usually negative" perception of evaluation; and 
the "'true believer syndrome,'" i.e. the fee'ling £hat, as one director 
said, "'We«are successful even if we can't show it on paper'" (p. '38). 
The survey described in King and Thompson- (1981) found that only 28% 
of the LEA users responding reported that the program effects they 
most care about can be directly measured; well over two-thirds (72%) 
felt that these effects can only be measured indirectly (43%) or not 
at all (29%) (p. 8). Users who feel that key program effects cannot 
be measured may find working with evaluators ^^frustrating experience, 
although it must also be noted that most of those responding reported 
infrequent contact with evaluators (p* 9) . 

Both this survey and a related simulation study have documented 
that administrators perceive the chara\^^pfc^ics pf different 
evaluator^ , In the survey, administrators were asked to categorize a 
typical evaluator in their district according to his or her technical 
skill (good or below average) and political skill (good or below 
average). The four' combinations of thqse variables create a typology 
of evaluators first suggested by Meltsner (1976). The local u^ers 
labeled their typical evaluator as follows $ 

Good in technical skill', good in political skill 49% 
(entrepreneur) 

i Below average in technical skill, good in political 26%' 



skill (politician) 
■ . j 
Good in technical* skill, l?elow average in political 17%" 
skill (technician) \ 

Below average in technical skill, ielow average in 8% 
political skill (placeholder) ' 

The recognition of evaluators" political skills in LEA settings is 

evident in that 75% of the respondents labeled their most common 

evaluators 'as high in political skills. The relatively equal 

importance of technical ski ^ s is reflected in th<* 60% rating for 

those categories with high technical skills. These* results contrast 

in part with the findings of Thompson (1980) in which the evaluators 

r* 

in an urban district evaluation unit were identified as either 
entrepreneurs or placeholders (p. 63). However, the second stu^y used 
a factor analysis of evaluators" reactions to statements to label 
them, and the difference in subjects and method may^ well account for 
th£ difference in the results. 

An additional study by Thompson and King (1981)* using simulated 
evaluation reports found that administrators may implicitly recognize 
differences among evaluates when they reacf reports. Given reports 
that varied in the use of technical and political language, principals 
responded differently. The results suggested, however, that adminis- 
trators were more attentive to the technical merit of reports than ttiey 
were to the evaiuatox's political sensitivity. Although the results 
. of any simulation study must >e considered tenuous, these findings, 
coupled with the work discussed above, suggest that users do perceive 
differences amorfg evaluators. To the extent that perceptions affect the 
subsequent use of results, they may be important variables to consider. 
As the results described previously suggest, the answer to the 
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second questiori — what are the characteristics of local evaiuation 
units and of local decision-makers? — is highly complex* There are, 
„ however, at least two generalizations to be made* First, local 
evaluation units are typically hot involved in program improvement; 
&nd second, although some research supports the claim that local users 
rind evaluation information helpful, other data su9§*st that these 
users may feel that evaluation is not of. value to them/\The picture 

that emerges of ,the local evaluation context, then, supports it as a 

• * \ * 

setting for potential non-use. / 

3. The third question is, finally, the key to improving 

the use of evaluation information at the local level. By knowing 

the factors that affect use, evaluators and decision-makers 

together can work toward increasing the appropriate use Of 

results. The discussion of what empirical research has learned 

about these factors will be separated "into two sections, 

presenting first those factors within the evaluator's control and 

second those factors pertinent to a local evaluation context, but 

■ 

out of any one person's control. 

Factors Evaluators Can Control . . Two variables that have 
received considerable discussion in the evaluation community are 
the technical quality of evaluation studies and of evaluation 
* reports. The Joint Standards contain both methdolog ical. 
Standards ,(D3, "Described Purposes and Procedures 1 *; D5, "Valid 
Measurement"; D6, "Reliable Measurement" ; D8, "Analysis*of 
Quantitative Information"^ etc.) and explicit standards foj: 
^ ^re]borts (A5, "Report. Clarity" ; A6, "Report Dissemination" ;" A7 , 
"Report Timeliness") Because' the need for the highest quality 
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evaluation methodology and reports is intuitively obvious* little 
empirical work' has been conducted on these topics. However, as 
Weiner, Rubin, and Sachse (1977) point out, to say that 
evaluation results are underused because of their low technical 
and scientific quality is to display a "faith in rationality,"' 
rather than an awareness of "evidence concerning the factors 
influencing the utilization of evaluative information" (p. 4). 

The results of three studies suggest that the use of 
evaluation information is' not directly related to the 
methodological quality of the evaluation itself. Patton et al. 
(1977) write that ". . . there is - little in our data to' suggest 
that improving methodological quality in and of itself will have 
effect 2D. increasing the , utilization of evaluation research " (p. 
151, emphasis in*or iginal) . Dickey (1981) Similarly found that 
methodological sophistication" (e.g. hypothesis-testing statistics?, 
multiple data sources, control/companion groups) had no 
relationship to the level of use (p. 73) . As mentioned above, 
while the evaluators interviewed in Kennedy, Apling, and Neumann 
(1980) worried about the methodological quality of their work, 
the users of their results were more concerned with the 

educational quality" than with the technical quality (p. 111). 
Webster; and Stuff lebeam (1978) reported that the evaluation 
departments of the districts they surveyed did have highly 
trained, methodologically competent evaluators (p. 32) . Evidence 

'j 

suggests, however, that implying the technical quality of results 
will not necessarily. improve Jthe level of their use. Not being 
methdological experts, decision-makers will typically use or not 

« * ~x 
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use evaluation results for reasons other than tjheir technical 

r 

* \ 

merits. 

The evaluation report is a second factor affecting use that 
evaluators can control. Several studies have examined aspects of 
the effect of reports on use. A series of studies by^lJra^Kamp, v 
^jBfown, and Newman (see, for example, Newman, Brown, and Braskamp, 

1980; Braskamp, Brown, and Newman, 1981) has systematically 

— — — — - — - < 

investigated the "relationship among the characteristics of an 

evaluator, an evaluation report, evaluation audience 

f v 

characteristics, and audience responses* (Newman, Brown, and 

Braskamp, 1980, p. 30) , using communication theory as a 

conceptual framework. Their results suggest that several report 

variables affect the use pf evaluation information. 
A 

4 7 a. The message source — the evaluator — "generally affects 
an audience's reactions not only of their ratings of the message 
source (evaluator) but also affects their ej^tent of agreement 
with the ^valuator's recommendations" fBraskamp, Brown, and \^ 
Newman/ 1981, p. 6). Both the title and sex of the evaluator can 
affect audience reactions. Even though readers read identical « 
reports, they rated those they thought written by a "researcher" 
as significantly more objective than those written by an 
, "evaluator" or a "content [art] specialist." Rpport readers were 
less likely to agree with reports which they thought were written 
by female evaluatorfi*when the field differed from their own? in 
their own field, they were. less critical of the female's results, 

although they were still more likely to agree with the male 

f i * 

„ ■ evaluator 's recommendations. 
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■ b. The message content can also affect readers* reactions. 
The use of jargon and data can affect audience ratings of 
technicality and difficulty. In one study, the report rated most 
'difficult by readers included jargon, but no, data support. 
"Generally, reports containing both jargon and data wer$ rated 
more useful Regardless of whether the readers were professionals m 
in the field or were lay persons from another field" (Newma*i^ x * 
Brown, and Braskamp, 1980, p. 33) ? Rated next highest we*e 

» 

reports lacking both jargon and data use, suggesting ari 

'4. ^ 

interaction between the use or non-use of jargon and -data in the 
reactions of readers. * * 

* A study by Brzezinski (1979) rated evaluation reports using 
the criteria outlined in the Joint Standards . The studies ranked 
the highest were "those that were the longest an^Ttadst research 
,repo**t~sounding" (p. 6). This suggests that the Standards 
•criteria, which give high points for appropriate technical 
language ana data, may then lead to reports that .users find most 
useful." To the extent that local users also' want as much 
information as possible (Alkin et al., 1974),, these technically ^ 
sound reports may also be the most effective. * m 

A worH of caution should be added here, however. The 
variables 'discussed — report use of technical language and data, 
and report length*r have not been exhaustively studied. . Other 
variables, for example', program, innovativeness or the inclusion 
of negative results, may well affect local use of , evaluation 
information. The finding of Brickell et al. (1974)^ concerning r 
the length of desirable reports in the federal bureaucracy ifiay^Te 
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equally true at the local level: -\ 

» ♦ 

* ' * " ' - * 

It was not unusual for officials to request shorty . 

reports f6r themselves and loager ones for their \ 
, subordinates-, but then to .find that their subordinates 
when- interviewed wanted shoijt^^eports for their own 
use ,»nd suggested longer reports for their 
. subordinates arm so on down the hierarchy (p. 60) / 



^ Further/ other research supports the view that 
administrators prefer qualitative information over quantitative 
information. Alkin's (1980b) yel^^own naturalistic use studies 
yielded the conclusion that "little evidence was founcPln the 
_^se studies i:hat research rigor was an' important , factor 

affecting utilization 11 (p* 24). Simulation research by Brown and 

" # 

Newman (in press) is ev<en more dramatic: 

Th^c simple addition Of an inferential "statement, such as 
"these differences were statistically, significant at the 
.05 level* n however ^resulted in lower (levels of 
agreement (with policy recommendations);, -In fact, for 
three of .four recommendations, the inclusion of the 
inferential statement resulted in levels of agreement 
lower than in the no data [experimental] condition. 

However, it^fs important to note. that the use of data does interact 

with other resist featur^s/in determining audience reaction (Brown, 

Newman, and Rivers, 1980, p. 72), so a simple interpretation of 
$ o ■ , 

• * * f * 

_ these results is not possible.' 



The effect of report timeliness ^seems, like technical 

• quality,, intuitively obvious; insults that are unavailable. to a 

•decision-maker at the time of a decision cannot be used. Randall 

;(1969) graphically portrays. this situation: 

Thtfxe is a.timeworn and oft-recurring spectacle of the 
- frantiq but finally productive researcher-evaluatOr, 
who rushed into the ex^^tive offices with, his data 
%nalj^is in han£, only to* find that the executives, 
several months previously, had made the important 
decisions that locked up the monies' and committed, the^ 
organization for the«ensuing months ahead (p. 1). 



The resultf of Dickey (1981, p. 73) provide .support for this /. 

assumption. Howiyer, two studies provide evidence, to the contrary. 

Pat-ton e t t al.. <1977) reported that timeliness w^not a critical 

factor in the use of federal health evaluation results (p. 15*0) , 

and Kennedy, Apling, and Neumann (1980) , studying local users, also 

labeled timelines? of questionable importance: 

Although we found several major decisions in the x 
making, the importance of timely data was hard to 
gauge* . . [E]ven when decisions were based on formal 
studies, timeliness was not critical (p. 13 1) . 

They note elsewherev that 'fior project directors, the results of. 

• formative evaluations ^are not available quickly enough; action 
must be taken using available data, even if it is last year's 
summative study. The case studies in Alkin, Daillak, and White 
(1979) also demonstrate the problems of reports that are made 

^available only after the next year's planning cycie. 

ThQ effect of report timeliness is, then, uncertain and 

■ 

perhaps even, in Cronbach et al.'s opinion, "much-overrated" (p. 
63) / Cleanly, decision-makers in local settings us^e information 

they are interested in as soon as the data are available in any 



form; as Patton. (1978) notes, the formal, report of an evaluation . 

* 3 

will hold no surprises for an alert decision-maker (p. 205) . To 1 
suggest that report timeliness is an important factor is to 
suggest a form of instrumental use**" But as was noted above, the 
instrumental use of evaluation information is, in general, 
limited. * Although timeliness may be* important for short-term 
instrumental use, it may not be as important* for the more general 

m 

use'*of evaluation results (Young and Comptois, 1979).. 
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c. 'The characteristics of the jreceiver can-also affect the 
use of information. Ratings of the usefulness of the Results of 
an external evaluation differed depending on the -organizational 
position of the audience. * Other important variables were the 
Level t>f"prof essiohal experience and the field o. the reader. 

"W * ^^^^ 

Involved here, too, J^the audience's perceived nee J for 

evaluative information in a particulaj: area (Brown, Newman, and 

RiVaxs, 1980) . H Those with a high perceived need for evaluation 

agreed more with tpe evaluator and were generally more satisfied 

with the information they had available than those with- a low 

perceived need" (1981, pp. 11-12). * 

A finding of King and Thompson (1981) suggests the 

importance of the evaluation report to the local user. Almost 

two-thirds of both principals and superintendents ranked written 

evaluation reports "which identify difficulties and discuss 
* 

possible actions to correct them" as tfre mqst * important form of 
evaluation reporting in their systems. To the extent that 
evaluation reports can be improved in meaningful ways (e.g., 
including appropriate data, gearing reports to certain 
organizational positions), empirical evidence suggests that the 
,use ^f results may then also improve^ 

Factors Related to th£ Local Evaluation Context . While the 
methodological quality of evaluations and the evaluation report 
are factors that local, evaluators can work to control, other 
factors affecting use flow from the context Qf evaluation. The, 
factors in this category that have?t>een empirically investigated 
can 'be divided into two topics: the issues local users face and 

57 * 



the process' o£ local evaluation use. ' 

The interviews summarized in Kennedy, Apling, and Neumann 

*** « 

(1980) mark an important addition to the literature because th^y 

provide evidence that the non-use of^information may in*many/ 

cases be beybnd the evaluator's control. The four categories of 

issues the users described — district-widfe, program-letfel, 

building-spebif ic, and clinical — suggest the variety of ways 

informatibj^JLs used at the local level. The use of evaluation 

information is discussed in all^but the clinical chapter, which 

dear? only with the use of test results. As seen in Tabl$ 1, the 

types of issues faced at different organizational positions vary 

radically* Dist;rict-wide issues are reactive, rather than » ' 

pro-active, created by the changing conditions^he institution 

faces. The issues program directors face often require increased 

personal advocacy; in such cases, personal information can often 
V 

obviate the nee'd for more formally collected data. Building- 
specific issues involve building management where little > • 
systematic information is used in implementing district policies. 

The summary paragraphs at the end of the three chapters 
describing these issues are thpught-provoking. 

^Following Chapter 3, "District-Wide Issues": 

, These findings raise some interesting questions about the 
use of evaluation and test data. . . [0] ne* wonders whether 
an evaluates {or anyone else) could ever predict what the 
issues would be insufficient time to generate data that 
would specifically address . . . [the unanticipated] issues 
[that characterize district-wid£ concerns']. . . If [the 
currently available descriptive statistics are appropriate 
to district-wide issues]. . ., then ^the role of evaluation 
studies per se — that is, assessments of the merits of 
^current practices — may be doomed to be a small, one, for * 
^'questions of merit seem to take, a *back seat' to questions of) 
new needs that must be' addressed (p. 45) r . ' 
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Following "Chapter 4, ~ "Program Issues": 

The situation program directors find themselves in raises 
some important questions' about the role that evaluation and 
testing can play in. program improvements. If, for example, 
we accept the fact that program managers, out of 
organizational necessity, must use data for advocacy, 
should we also accept the fact that such uses will probably 
also entail distortion of the findings? How does one draw 
the line between appropriate and inappropriate uses of data 
* in advocacy situations? Furthermore, data tend to be used 
in group settings, that is, a program director discusses 
its implications with one or two trusted staff members or 
an evaluator and then uses it to persuade others. The 
process has many layers and is diffuse. How would one 
i intervene to improve it? (p. 65) . 

, Following Chapter 5, "Building-Specific Issues": 

These findings are significant in that a considerable body 
,of literature suggests that building princtipals are key 
contributors bo the quality of education in their schools. 
Yet, for -the most "part," the issues they face are' not" issues 
that could be identified, clarified, or solved via 
» systematic data (p. 82) . 

- In other words, thafruse of evaluation information at the local 

V 

level may be severely limited because of the kinds of issues 
users face at this level. The question for evaluators then 
becomes one of determining those issues and situations that can 
benefit from data collected systematically and of working to 
insure »that th<T information is presented in useful ways. To do 
this effectively will require a thorough understanding of the 
process pf evaluation use at the local level/ the second set of 
use factors related to the context of evaluation. 

The many variables affecting the process of local" 
information use have not been documented empirically. In their ' 
study, Patton et si. (1977) found only two. factors significantly, 
related to the.use*of evaluation -results: a "political •' 
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considerations factor" and 'the "personal factor* (p. l49)/i.e. 
tt}^H?resence of at least one ^>erson in an evaluation study who 
cared about the process and using its > results ? In an LEA 

* 

setting, Granville's (1977) simulation^ study similarly found that 
the "political"" and "social influence" factors significantly 

affected principals* decisions. Granville's social influence 

«- 

factor indicated the importance of the professional peer group at 
the local level. These results suggest the obvious:' to work in 
a school district is* to live in- a political environment' where 
^^Spe^if-ic^individuals can make a difference and can actually make 
evaluation use occur. 

The conclusions of empirical studies fcave^r^p^atedly 
recommended a collaborative approach to l^ocal evaluation efforts, 
i.e. evaluators and decision-makers working together to create 
and use evaluation information. Both the political- and personal 
factors suggest the validity of such an approach. Based on the 
study of Title IVC project directors cited earlieV, Dickey (1981) 
concluded that " evaluators should adopt a more collaborative 
role , involving the decision maker and the staff in decisions 
about the evaluation " (p. 76, emphasis ' in ^original) . She also 
writes that "evaluation, conducted at a distance, is less likely 
% to be valid and unbiased and thus,* less useful?* (p. 76} . 
Goldberg (1978) found that New ¥ork City'ESEA 5 
administrators received helpful information"* from evaluations in 
the problem awareness step. of decision-making, but not in the 
later steps, i.e. in finding possible courses of action or in 
choosing among alternative courses of action .{p. 19) . He 



recommended collaboration as a way to increase the use of 
evaluation information throughout the decision-making process. 

Two case studies of Alkin, Daillak, and White (1979) 
provide graphic evidence of how collaboration can help the_use 
orocess. In the Bayview example, the "evaluator, assigned by 
chance to a project, . . . [became], in a very shct period of 
time, a trusted advisor to decision-makers who . . . [had] little 
predisposition toward evaluation" (p. 208) . The contrasting role 
of the evaluator in the Valley Vista example shows how an outside 
evaluator intent on collecting mandated information can force 
participation against the local staff's wishes — but not use. 

"Collaboration" can be as straightforward as having 
evaluators share results wi-kfe decision-makers as the. .evaluation 
progresses. David (1981), for example, states that "the 
provision of feedback, particularly when explained in person, 
provides what may be a necessary but not sufficient cpndition for 
utilization of the evaluation information* (p. 29) . In their 
survey of evaluation offices, Lyon et al. (1978) noted that most 
reported feedback to project personnel consisted of "informal 
verbal recommendations rather than formal written ones" (p. 87) . 
Further, "the importance of verbal, rather than written, 
evaluations wa? also underlined in fieldwork activities" (p. 43). 
It is possible that. this degree of collaboration has .helped local 
decision-makers to apply evaluation results in the subtle ways 
now recognized as constituting use. 

Although research thus suggest^ that collaboration may be a 
good idea, ^the constraints of practice may limit this approach. 
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The evaluators* interviewed in- Kennedy, Apling, and Neumann 

(1980), for example, . knew that extensive involvement in a • 

program — becoming "close enough to program directors. to act as 

their personal consultants" — would influence information use, but 

they knew too that it was a "role that required more time than 

* 

most of them were able to spend" (p. 117) . Kennedy et al. give 
three reasons that may limit t-he efforts of evaluators at 
encouraging use: (1) some people already use information 
effectively and the effect the evaluators could have had is 
unclear; (2) others had not been converted to use despite the 
presence of evaluators who "ppssess an imposin^array of personal 
skills and qualities"; and (3) much of the use of data "occurred 
because of ofch^r, institutional pressures" beyond the 
single-handed control of evaluators (pp. 118-119). The role of 
professional knowledge in evaluation use, for example, has not 
yet been carefully studied. 

In recording that two-thirds of the responding evaluation 
heads felt that their personnel resources were inadequate, Lyon 
et al. (1978) provide another reason for 'questioning the 
probability of the, pract ical ity of the collaborative approach. A 
^further reason the approach may be unsuitable stems from evidence 
presented in Smith (1980b) that merely 'increasing user 
participation in an evaluation study may not increase the use of 
the recommendations (p. 64) . Whether board members' involvement 
in accreditation studies parallels other local users' 
involvement, however, remains to be seen. 

* Like the \second question, the answer to the third 

r 

" 49 



question—what factors affect use at the local level? — is' complex 
and, to date, not clear.' Certain factors are known to be 
important in the^ process of local evaluation use, and research 
has suggested the efficacy of a collaborative approach. However^ 
^knowledge of the use process remains limited. 

Strategies for Improving Use 

* Although few empirical studies* of the local use of evaluations 

exist, frequent mention is made in the evaluation literature of 

strategies for improving such use. Cronbach et al. (1980) argue that 

"if any single intellectual sin is to be blamed for the present chaos, 

it is the readiness to make general assertioas that supposedly afcfily 

to all evaluations" (p. 51} ♦ Concurring, Patton (1978) warnar t* at 

s The overall problem of underut ilization of evaluation 
research will not be solved}.- by r compiling and following 
source long list of .evaluation proverbs and axioms. Real 
world circumstances are too complex and unique to be 
routinely approached through the application of isolated 
pearls of evaluation wisdom (pp. 19-20). * 

The following, therefore, will be a general discussion of the factors 

identified in the literature as affecting use and 'strategies for 

addressing these: The discussion will again be divided into those 

factors that are more or less within evaluators' control and those 

more related to* the evaluation context that cannot be controlled, but 

must be dealt with. - 

Factors Evaluators Can Control . Although method ologi cal quality 

has not been shown empirically to be either necessary or sufficient 

for use to occur, common sense and the Joint Committee's Standards 

suggest that it is a minimum criterion for good evaluation practice. 

Alkin (1975) points out that "apparently, one of the best- 

\ 
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conscientious defenses against non-utilization of evaluation findings 
is a technically sound, methodologically credible study" ^p.. 207) ; 

/ without such defenses, critics can easily dismiss evaluation results 
by attacking their technical credibility (p. 206.) . Somewhat 
idealistically, Leviton and Hughes (1979) suggest, "If 
[methodological], quality does influence use, it is likely to do so 
primarily through increased trust that the findings are an accurate 
. picture of the program* (p. 25). While it is true that some 
administrator disdain for empiricism niayb/ appropriate because 
quantitative forms of representation "inherently are insensitive to 
some of the significant aspects of classroom life" (Eisner, 1980, p. 
11), it is also true/ that .good methodology will^ insure results that 
are worth usij^/"The central message in this regard," however, "is 
~ w that it is not enough to conduct methodologically sound research" 
(Johnson, 1978, p. 12)* 

In addition to sound methodology, as noted) above, a second 
factor within the ^valuator's control, is the evaluation report., in 
both its oral and written forms* Empirical results have demonstrated 
the varying effects of the message source, message content, 
and — perhaps — its 'timeliness. The importance of effective 

, communication by evaluators has been Tiighly emphasized in 

non-empirical writing. The likelihood of the evaluation report itjself 
having jjpmpact may be small because, as Cronbach et al. (1980) point 
out, "T^Hpvaluator's final, formal report is essentially an archival 
document, not a live communication" (p. 185) . However, given the 
local users" interest in having such reports, evaluators must work to 
make them as good as possible. "Most of these reports," writes Denny 
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(1980), *ha.ve ofie omnipresent feual ity : they are dull, dull, dull" (p 
4) 5 Datta (1979) reports that) "Although titled 'evaluations,' more 

accurately . . . [the evaluation reports she examined] might be 

/ 

presented as descriptive statistical accounts of some aspects of 

educational programs 11 (p. 17) ; las such, they may not be useful to 

) ' ' 

local decision-makers. 

Suggestions. for improving reports include using details of 
events and people, making the evaluation report a "well told story" 
(Denny, 1980, p. 5); including details of the evaluator's personality 
i.e. his or her competence, belief, style, etc, (Denny/ 1980, "p. 6); 
using executive summaries on different colored paper (Alkin, 1975, p. 
208) j and using appendices to hold more technical information (Alkin, 
1975, p. 209). Datta (1979) also recommends the- executive summary 
approach, i.e. that evaluators "address the few important questions, 
present the evidence, and state the action implications in one page o 
less" (p„ 23). Brzezinski (1979) points out, however, the limits of 
such summaries for truly informed decision-making (p. 1). 

Should evaluation reports contain recommended actions? 

Empirical research has not provided a definitive answer to this 

question. Although the answer is clearly tied to the evaluator's 

conceptual approach to evaluation (see Webster and Stufflebeam, 1978) 

to the extent that such recommendations may assist harried 
* * 

decision-makers, they' may well be of value. Haller (1974) observes 
that • 

Evaluation problems concern decision, decisions presume the 
existence of alternatives, and so the purpose of evaluation 
is to ,help delineate alternatives and to provide 
information to help decision makers arrive at more rational 
choices (p. 403) . 
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However, Weiss (197'9) notes that many evaluators are not policy 
oriented; "evaluators do not always — or even often — come up 'with data 
that' give explicit guidance for action" {p. 3). Zepeda (1980) / 
suggests that this situation is n<pt entirely satisfactory: "Informing 
local decision makers that the local Title-I program is not effective 
does not give them the information they need to improye it" (p. 1) . ' 
Will administrators perceive recommendations as an unwarranted 

intrusion into the policy arena? They will if the evaluator "*s^ 

v 

offerings take the form of grandiose schemes. They may not if 
specific policy alternatives are mentioned and the evaluator merely 
presents objective evidence, both for and against, regarding the best 
predictions* about likely program impacts. Evaluators sometimes , 
perceive administrators as defensive of their territory, and in some 
cases these perceptions are fully justified. But many administrators 
do not feel threatened by good ideas and will take their wisdom 
wherever they can get it, especially if they believe that the 
recommendations are being offered ij^a^Tincere attempt to help rather 
than in an insincere attempt to be Ma<;hiave N lli^ 

and Brown's (1980) results indicate, the utility of this effort will 
partly be determined by the situation-specific personalities and the 
needs of the involved administrators. 

The literature suggests that the informal contacts between 
evaluators and users are more impprtant than written reports in 
effecting use. Four of the six themes discussed in the New Directions 
Sourcebook entitled Utilization of Evaluative Information (Braskamp 
and Brown, 1980) ( point to the importance of this interaction: 
utilization as an "immediate concern -of evaluators* ; the "active role 




of evaluators in enhancing utilization"; the need for a "highly 
interactive consulting relationship between evaluators and key 

decasion-makets",; and the "importance of the communication process^ 

* * » * » 

(pp; 91-94) . Consider also the following quotations. 

The evidence' on dissemination suggests that informal 
. , communication that cuts the red tape may enhance 

utilization, although quality of information may sometimes 
v suffer and dissemination will be haphazard (Leviton & 
Hughes, 1979, p. 21) . 

For while information is an essential resource for decision 
makers, the manner in which it is converted into policy is 
based as much or more on interpersonal, organizational, and 
psychological factors than on the actual information itself 
(Guskin, 1980, p. 45) . ' 

Utilization is usually the result of the relationship 
between the evaluator and the user more than anything else._ , 
If the user knows and respects the evaluator, utilization** 
has its -highest potential (Holley, 1979, p. 8). 



Much of th^r most ^significant communicat ibn i$ informal, not 
all of it is deliberate, and some .of the largest effects 
are indirect (Cronbach et al., 1980, p. 174), 

The message for evaluators seems clear: informal communication 

with the information users must be stressed throughout the evaluation 

process if results are to be used. It is no^ always possible to 

anticipate when information. will be needed in the service of 

educational decision making. In fact, as Brickell, Aslanian, and Spak 

(1974) note, at the federal level, the administrator "can never know 

when he will need it [evaluative information] . The process of 

government decision-making is not so orderly or regular that he can 

schedule his need for information" (p. 24) . Kennedy) Apling, and 

• * 

Neumann (1§80) provide evidence that the same is true of local 

administrators who deal with district-wide issuers". 

Wise (1980, pi 15) and Cronbach et al. (1980, p. 160) label the 

t ■ ' • 
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evaluator an educator whose job it is to teach bthers, about the 

* ♦ * f 

t s 

program studied. The information presented bust be « 

• • . Understood, credible, and coherent to the intended 
audiences and the evaluator must be a guide: for politically* • 
feasible -action. . . [I]f a two-way cpmmunication channel 
between the evaluators and users is established, provisions' 
for trust and mutual problem solving are, more likely (p. 
11). . . x 

Evaluators are, then, in many ways responsible for making use happen.^* 

Lee and ,Ho/ley (1978), give advice on how to disseminate 

evaluation results. Included among the practical tips are such things 

as training the press to properly interpret evaluation data ar^d Using 

J 

audio-visual aids during oral presentations. In more theoretical 
paper, Holley (19B0a) suggests that evaluators, knowir/g the political 
nature of evaluation settings, should even use potential use as a key 
criterion in the allocation of evaluation dollars. It is better, in 
her view, to attack difficult "mountains" that users are interested in 
than to work on smaller, more methodologically tidy-^^olehills" where 
results may not -be used. VAlkin (1975, 202*T*^nd Bosco (1971, p. 70) 
suggest th,at potential and intended uses should be specified e^rl$ in 
the evaluation process. 

Factors Related to the Local Evaluation Context s It is not 
possible finally^ to draw a fine line between utilization factors that 

N 

evaluators can control -*and t^ose they cannot. Even those that are 
"most controllable — the methodological and report quality discussed 
<above — can clearly be affected by the complex interactions involved in 
an evaluation setting. The practice of "fuddling," i.e. "allowing- 
evaluation results to disappear in a ream of paperwork, excuses, and 
mea-ninglesa exchanges" (Davis and Salagin, 1975, p. 623), can limit . 
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the effect of even the. finest results, character istics^bf local 

programs and decisiort-mak6rs are , an important paj^ o £ - l he local 
evaluation content, and^empir ical .studies have suggested that certain 
characteristics af feet* the^u^e of " evaluation information. Other- * 
writings support this~view/ N_ . \ 

Certain "prcjgram controlled factors" work against use in any. 

' / • - . 

evaluation context^ (Alkin, 1975, p. 2Q7) . Holley (1980b) refers to 

"Required Losing: Factors That Prevent the Use of Required Evaluation 

Reports at the Local Level," She gives two such factors: {if the 

requirement that mandated reports include lists of numbers "that are 

in themselves devoid of meaning"; and (2) an emphasis on - object ives 

that may be good for planning, but inappropriate for. evaluation (pi 

107), Grobe (1978) adds three additional factors that work agaiti^t 

effective use: (1) the fact that local decision-makers are unfamiliar 

with data; (2)- the higfc^, emotional involvement of - local project staff; 

and (3) the iQcal expectation that precise answers come out of 

evaluation only, infrequently (pp. 2-3^* 4 

Another factor affecting use, as Granville (1977) and Dickey 

(1981) demonstrate, is the age, innovat ivfcness (H.e. routine or 

novel), and/or maturity of a *prograifu Brown (1973) explains, J . 

Thet que stion "as to when "and which levels of information 
school management need^s- depends on the lengttr of time a 
* '.program has been, in, operation arid the degree to which that ^ 
-^project Oyerf&ps other programs within the school system 

(p. 2) . *" . v *- - ' , " . 

' a fc 4 » N 

Cronbach et al. (1980) discuss four 'stag.es of program jnaturity that 
apply equally well tcT local and national programs: the. breadboard 
sta^ejv the super real i2at ion; tHV prototype; ,^nd the established 

program (p, 108) , ^Depending on.4fhe program's maturity, the use of - 

... 9 



evaluation ifeEogasation may .differ, and strategies for improving use ^ 
must consider., this. "Most of the time," for example, "the fully ^ 
established, mature program is allowed to gc* its own way" {Cronbach et 
al., 1*80,. p. 113); Kennedy, Apling, and Neumann (1980) write that at 
; the local leyel," fully developed ^programs are rarely eliminated, but 
instead are simply 'not -expanded .(p. 60) ^ 

The information user's organizational position is another factor 
affecting the type of information and appropriate use strategy. 

4 

Leviton and Hughes (1979) ppint to a distinction between the use of 

evaluation for policy decisions and fog programmatic decisions (p. 

12) , and Brickell, Aslanian, and Spak (1974) write ^ 

Those officials who are in a position to control the 
project from day to^ day or month to month and who are 
Responsible for exercising such control have a* far greater * ^ 
interest i* monitoring reports than decision-makers at 
higher executive levels (p. 59), 

Furthermore, evaluates should remember that "research is often most 

useful to those wjho do notJhave the authority to promote a policy, 

i*e., teachers" (Hamilt6n,*T?980, p. 7) • 

The empirical research cited above suggests two additional * 

factors critical to. the use of evaluation information: the political 

considerations 'factor and ttffe personal factor. Although many * » 

evaluators may still feel uncomfortable admitting it, evaluation is 

inherently a political activity. As Cohien WL972) writes, "To the 



Co hie n (AS 

s< ■ 

neTTtr* has: 



extent that information is afi instruments basis, or excuse for 

changing power relationships within or among institutions, evaluation 

is a political activity" (-p. 139), The implications of -this fact are 

.identified by Dickey (1979) when 4 she writes 

Judgment has an awesome ring, and it is not surprising that 
those who are being judged feel anxious, 4 even threatened. 
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Add to this the N political cont^t in wKicfi thje process 
takes place (and there is* always a poljJ^i'cal context) and 
we have all the ingredients for dysfunctional communi- 
cation — high Levels of stress leading to" comiflunication 
patterns arising from individual defense mechanisms (p. 3). 

Certainly the political considerations factor, whose existence 

* 1$ supported by empirical evidence, can help or impede the use ,of * 

r , ■ / 

results. As Cronbach et al. (1980) have written, "No*matter how 

excellent a study is technically, its facts will not' sweep political 

sentiment and power aside", (p. 46). This does not mean that 

evaluators must themselves engage in political machinations. However, 

as r Meltsner (1976) suggests, the effective evaluator "tries to 

understand political considerations and then to make them an 

integrated and expli^t^pa^t of his analysis" (p. 43). According to 

Brown and Braskamp (1980) r "This means that the relationship between 

the evaluatpr and key program staff, and the evaluator's understanding 

of the organization in its internal and external political ' ^ 

environment, are critical, for successful utilization" (p. 93) . This 

suggests that evaluators must work at understanding the politics of 

their agencies and- attempt to meet the political needs of 'involved 

persohs whenever doing so will not jeopardize the integrity of the 

evaluation. . - ^ 

In addition to the political considerations factor evaluators 

y ' " 

at the local level must take advantage ofi the personal factor. 

Leviton and Hughes (1979) point out that the impact of individuals on - 

organizations is rarely great and suggest two reasons for the effect 

of the personal factor: (1) in smaller sitings individuals may be 

able to make a difference; and/pr (2) in larger bureaucracies th£ 

* * 
difficulty of effective communication may ajlow individuals the 

• />'" ' ' 
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opportunity to make a difference (pp. 27-28). Whatever the cause, the 
effects of the personal factor should not be ignored by local 
evaluators. 

Qn*-4mportant way to go about" this is by targeting the 
evaluation toward identified information users (see, for example, 
Patton, 1^78). As Johnston (1978) writes, 

There is a sort of ecology for each educational program, a 
network pf people in different* roles who influence (or are ' 
influenced by) the outcome of the program being 1 evaluated. 
If this is true, and research utilization is the goal of 
the evaluator, then there are multiple audiences for an 
t evlauation, not just the decision-maker who commissioned 
the evaluation. So the ev'aluator has a first task of 
identifying who these other actors are (p. 1). 

* 

v^-3*he variety of issues raised by users in Kennedy, Apling, and Neumann 
(1980) certainly supports the need for this effort on the part of 
evaluators. 

However, it was noted above in discussing the difficulties 
inherent in collaboration tjtet the effort can be frustrated b/ the 
complexity of the organizational network. As Randall (1969) explains, 
"Typically, the ^decision process in an organization involves a complex 
network of persons who have varying degrees of influence om the one * 
who may'hae constituted authority to make 3ny given decision" (p. 7). 
The situatipn is further complicated because, as .Granville (1978) 
notes, an y evaluation study "has to persuade not just the people' who 
ostensibly make the decisions, but also the people they have to 
persuade" (p. '29, emphasis in original). Thus, Alkin and Kosecoff 

1973) conclude that *ident if ication of the** program's decision 

* • t * • ■ 

maker [s] is perhaps the most ^lusive variable associated with a 



decision context" (p^. 3) . 
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Assuming that decision-makers are identified, non-empirical ' 
j> 

writings have strongly supported' a collaborative approach as building 

I 

on the strengths of both the political considerations and personal 

factors. Gray (1979), for example, has developed a procedure and 

m-'jpual for collaborative evaluation. Cronbach et al. (1980) suggest 

chat the evaluator "should engage others' in a collaborative attempt to 

understand social events and take appropriate action. Influence comes 

frcu engagement, not detachment" (p. 153). Patton (1978) has 

recommended the collaborat iy6 development of evaluation designs; "it 

is crucial that identified decisionmakers and information users 

participate in the making of measurement and methods decision so that 

they understand the strengths and weaknesses of the data — .and so that 

they believe in the data (p. 202). Ross (1980) has even suggested 

that evaluators and administrators should together specify decision 

rules in advance of program implementation, rules specifying what 

decisions will be taken if various evaluation results occut (p. 66) . 

Certain practical difficulties wifeh the collaborative approach 

have been discussed previously. Two others are suggested in the 

literature. First, what Patton. (19*78) has termed the "goals shuffle" 

can readily destroy efforts to estimate discrepancies between program 

goals and actual program outcomes: " / ^\ s ^/^i 

The goals clarification shuffle involves a sudden change in 
goals and priorities after the evaluator is firmly , 
committed to a certain set of measuring instruments and- to 
a research design. The choreography for this technique is 
quite Vimple. .The top priority program goal is moved two 
spaces to 'either the right pr left and four spaces backward 
' (p. 100). 

Second, Rossi (JL972) has argued that ^hat might be called "methodology 
shuffles*, can also occur if evaluation results prove unpopular: - , 
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It is easy to attack the methodology of any study: 

methodological unsophisticates suddenly become experts in 

sampling, questionnaire construction, experimental design, 

and statistical analysis, or borrow experts for the 

occasion (p. 229, emphasis in original) . • 

* « 

Collaboration with local decision-makers runs the risk of subversive 
activities. 

Can a collaborative approach to evaluation work in local 
settings? The a"nSwer to this question is a hopeful yes — maybe. As 
explained above, the evaluation literature has increasingly' recognized 
that the process of change in organizations is rarely abrupt or 
dramatic; and the instrumental use of evaluation results is therefore 
an unrealistic expectation. Evaluation's strength comes from its 
ability to assist in incremental accommodation; to provide needed 
information to decision-makers; to, in Patton's wt>rd$, reduce their 
uncertainties about the program that surrounds them U978, p. 31). 

Weiss (19^^ writes that the "use of evaluation appears to be 
easiest when implementation implies only moderate alterations in 
procedure, staff deployment, or costs, or where few interests are 
threatened" (p. 320) . In this' light, a collaborative approach be,twee*n 
evaluators and potential users makes theoretical sense, the evaluator 
assuming -the role of change agent (Davis and Salasin, 1975; Joint 
Compiittee," Standard A8, p. 47) or linking agent (Havelock, 1968; 
Hayraan, 1979; Bank, Snidman, and Pitts, 1979). "Assisting in gradual 
accomrnqdation, evaluation is both conservative and committed to 
• change (Cronbach et al., 19*80, p. .' At least part of the appeal 

r 

of this- approach" is its minimal requirements l Major organizational 
changes (see, for example, Weiner, Rubifi,' and ' Sachse , 1977,, for a 
t proposed alteration of the federal, evaluation structure) are unlikely 
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to k occur in local education agencies; in some cases collaboration no 
doubt already takes place; in others, all that is required will be a 
shift of emphasis. 

The evaluation literature contains numerous suggestions for 
making collaboration successful. Certainly the informal reporting 
procedures discussed above will encourage the type of communication 
essential for working together. Targeting evaluations and identifying 
appropriate issues will also help the collaboration process. Alkin 
and Daillak (1979) argue that "evaluators who concentrate on the 
mandated evaluation tasks run the very real risks of losing the local 
audiences" (p. 47) . This suggests that evaluators should concentrate 
evaluation efforts on the highest priority information needs of 
specific administrators, even if these needs^require work beyond that 
mandated by external' funding^ agencies. As Alkin, Daillak, and White 
(1979) note, "If -the evaluation addresses a pressing concern of a 
potential user, then the evaluation information is more likely to 
draw, and hold, the user's attention" (p. 238). Patton (1978) goes so 
far as to suggest enhancing use "by focusing on fulfilling one purpose 
extremely well, so that at least the decisionmakers' central questions 
are answered (p. 83) t . 

Since administrators are not always able to anticipate or 
articulate future information needs, evaluators "should anticipate 
questions and be proactive" (Law, 1980, p. 74) . Stake (J973) makes a 
/ similar point. 

The evaluator, I think, has a responsibility to snoop 
around and to guess at what decisions may be forthcoming. 
He should use these guesses to orient his evaluation plan 
. (p., 305). 




Gorham (1970) argues- that evaluators should "be clairvoyant about 
forthcoming issues" (p. 104). Cronbach et al. (1980) suggest that 
ideally the evaluator "interprets outcomes broadly and envisions 
possible negative effects as well as the intended positive ones" (p. 
260) . In' short, evaluators should identify some evaluation issues on 
the basis of emphatic and proactive anticipation of administrators' 
future information needs. If these anticipated needs do not arise, 
evaluators need not highlight the results of the inquiries that they 
initiated. In any case, evaluator .credibility should be improved when 
administrators sense a sincere effort to be , responsive, even if this 
anticipation is not always precisely accurate. 

To be effective collaborators, evaluators must also demonstrate 
to project personnel and administrators that they sincerely care about 
the needs of program staff and the program's clients. This is the 
other side of the personal factor and means that evaluators must try 
to not be threatening and authoritative; how evaluators comport 
themselves affects the psychological frameworks with which 
administrators interpret evaluative information. This conclusion may 
discomfort some evaluators who believe they offer objective truth that 
has intrinsic value independent of evaluator personality or approach. 
Nevertheless, administrators have their own paradigms for viewing the 
world — paradigms that are rational in their context — and evaluators 
must accept that the manner in which the evaluator interacts with 
administrators and staff with affect, in an important sense, the 
credibility of subsequent evaluation results for administrators. One 
strategy for avoiding conflicts is to prepare a formal memorandum of 
agreement prior to a proposed evaluation in which purposes and 
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procedures are clarified (Webster and Stuff lebeam, 1978, p. 39). 
Another is to be "active, reactive, and adaptive" (Patton, 1978). 

Evaluability assessment is one procedure with great potential 
for local evaluation efforts. Also known 'as exploratory evaluation or 
accountability assessment (Rutraan, 1980, p. 12), evaluability 
assessment is used to determine in advance the likelihood of an 
evaluation's success* Developed initially by Joseph Wholey and 
associates at the Urban Institute, evaluability assessment focuses 
first on program characteristics and then on the feasibility of 
conducting an evaluation study as planned. As Alkin (1975) points 
out, "perhaps it is important as evaluators that we learn tcf^^~ 
distinguish those situations in which the context and decision factors 
are so pre-d^termined that it can be inevitably said that no one needs 
it [i.e. the evaluation] and no one cares" (p. 211). At the local 
level, evaluability assessment can be a first step in the 
collaboration between evaluators and decision-makers and may result in 
more appropriate evaluations, and, hence, better use of tfeeir results. 

An evaluability assessment begins by examining characteristics 
of program components to determine how close they come to an ideal, 
asking if they are well-defined and capable of being implemented in a 
prescribed manner; if goals and effects are^clearly specified; and if 
plausible causal connections link goals and effects. In this stage, 
the evaluator develops three models of the program: first, a "program 

documents model," which shows the causal links described in program 

* - — — 

materials; second, a "program maimer's model," which modifies the 
first model according to information from key decision-makers; and, 
finally, an "evaluable program model," which presents the evaluator's 
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views of what components can appropriately be evaluated. This first 
step develops an explication and analysis of the theoretical premises 
of the program and a specification of the "process model" of the 

program— two methods suggested by Weiss (1972, p. 323) for increasing 

i ' 
the use of evaluation results. 

Because the purpose of the evaluation determines its 

• * 

methodological requirements, the second stage of the evaluability 

assessment, the feasibility analysis, begins by determining the 

purpose(s) of the given evaluation, then looks at the constraints on 

the evaluation to see to what extent the research requirements can be 

met. Considered in the feasibility analysis are program design and 

implementation, information requirements, and research design. Its 

product is a list, based on the evaluable program model, of, the 

N program components and the goals and effects to be studied in the 

ultimate evaluation. The program evaluation, when it is finally 

conducted, benefits from the limits set during the two stagee of the 

evaluability assessment; only what can and should be evaluated will be 

evaluated. As Rutman summarizes, 

Constraints inevitably limit the pursuit of the study's 
goals.* However, recognition of the practical implications 
of the constraints is crucial in determining the point at 
which the study's objective can no longer be achieved. At 
this point the purposes of the evaluation may have to 
change. Or further work may be needed to remove obstacles 
that inpede the measurement process, as a precondition of 
doing the study (p, 145) . 

Part of the appeal of evaluability assessment for local 
evaluations is due to the fact* that programs themselves can benefit 
from the process. Before the program evaluation begins and as a 
by-product of the evaluability assessment, program directors may make 

i ■ ■ 
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changes in the -program to enhance its evaluability . Strategies may 
include analyzing problems; specifying outcomes; assessing program 
design and implementation; and facilitating program development (what 
Rutman calls "formative research") . 'By helping local decision-makers 
in this process, evaluators can educate users to the practices>- of good 
evaluation while they themselves learn the details^ of the functioning 
program, ( 

The benefits of conducting evaluability assessments, then, fall 
into two ^categories. First, evaluablity assessments facilitate 
evaluation planning by establishing priorities, by providing 
"front-end control" over 'the evaluation .process, and by allowing. a 
wise allocation of evaluation dollars. Second, they facilitate 
planning and implementation by .providing information on appropriate 
directions both for information users and for evaluators. To the 
extent that internal evaluators are "captive ... in the sense that 
they do not often have a choice whether or not to do an evaluation" 
(Caulley %nd Smith, 1978, p\ 31), evaluability ^assessments may 
currently, however, have lfmited applicability. i 

The preceding review of both empirical and non-empirical 
literature has included a variety of factors involved in local uses of 
evaluation information and numerous suggestions as to what evaluators 
should 3o about them. No one method can be viewed as the panacea for 
evaluation's utilization woes, and it is important that evaluators use 

r 

a holistic approach to adopting these strategies. As Weiner, Rubin, 
and Sachse (1977) argue, "Attempts to increase evaluative influence 
which focus on a few of these factors in isolation and which do not 
recognize the highly complex and interactive system of forces 
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constraining ^evjluator activity are likely^ to fail to alter the 
overall effects of the system" (p. 23). 

A Critique of Previous Use Research v 
The previous 'research on the use of evaluative information has 
been dominated by two research approaches. The first approach is that 
of the retrospective case s.tudy, of which a limited number have been 
reported (e.g. Alkin, Daili/k, and White, 1970) . The second type of 
evaluation use study involves simulation investigations that at least 
purport to be grounded in .communications or attribution theory (e.g. 
Newman, Brown, and Braskamp, 1980, pp". 29-36). Simulation research 
typically presents administrators with a "simulated" evaluation report 
in which different report features, e.g. the sex of the evaluator, are 
varied, and the impacts of the variations then assessed. Both 
research approaches have made major contributions to. our understanding 
of the use of evaluative information, but both have recognizable 

weaknesses. 

' \ 

Alkln^(1979) has consistently argued that 

The forces which lead to utilization are* indeed complex. 
This complexity in combination with our current inadequate 
understanding of evaluation and utilization requires a 
methodological procedure sufficiently sensitive to capture 
the nuances involved — naturalistic research is currently a 
. most appropriate tool for a study of evaluation utilization 
(p. 13). 

^^lkin, Daillak, and White (1979) are less restrained when they argue 
that " the choice of appropriate research strategies can be reduced to 
one class is naturalistic research methods " (p. 3'2, emphasis in 
original) . Although appealling and often appropriate, this case study 
work can be criticized on at least' three grounds. ? 
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First, case study research pan no longer call itself theory 

generating. There is now theory, or at least the beginnings of 

theory, that can be relied upon in conducting use research. For 

example, our understanding of evaluation is reflected in fairly 

♦ 

elaborate conceptualizations of types of evaluation use {Britskamp, 
1980). As noted above, several researchers have differentiated 
instrumental, conceptual, and symbolic uses of evaluation* Meltsner's 
(1976) conceptualization of "types" of evaluators also could support 
theoretically oriented inquiry* Finally, communcation-r elated • 
theories^can greatly -enhance our understanding of at least the report 
•phase of the evaluation endeavor (Brown and Newman, 1979 a) . These 
various theoretical frames are certainly not yet fully developed, but 
further progress in developing theory absolutely depends upon testing 
and elaborating the constructs that are already at our disposal. Prom 
this point, forward, failure-to theoretically ground naturalistic 
research seems both unnecessary and unfortunate (Thc^mpson, . 1981b) . 

Spcond, most of the previous case study research has been based 

t> - 

on post hoc interviews with evaluators and' evaluation clients. 

Levi ton and Hughes (1979) have commented on the' dangers of 

retrospective research methods: "Given officials* faulty memories, 

retrospective research may be biased in favor of a few dramatic 

instances of uafe, rather than, frequent but modest ones" (p. 15) . 

Retrospective methods are economical, but their limitations must be* 

acknowledged, and futulfe studies should look to both the past and the 

present for data. 

Third, some case study research can be criticized for. th^Hy it 

has been reported. Although some authors -{Meltsner, 1976; Patton et 

s 
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al., 1977) have integrated themes and casQ study evidence into a ' 
unified analysis, other researchers have presented case study evidence 
in non-integrated, blocks of detail abs6nt of any themes* . Presented as 

* * t * 

unrelated case histories, such naturalistic research may represent a 
poor return on a necessarily extensive investment* Ironically, if .an 
evaluator communicated evaluative data as poorly as some case study 
research has been communicated, .the evaluative information might never 
.be used* 

The second approach to evaluation use research, the simulation 

study, can also be criticized on three grounds. First, although this 

) 

research is typically represented?^ being theoretically grounded in 

communications or attribution theory, this grounding has up to now too 

frequently taken the form of name dropping the theory's title without 

invoking specific propositions of the cited theories. Of course, in 

some simulation studies tested propositions have been implicit, but 

good research. practice would seem to dictate that even in these 

studies the proper primary focus of the work, i.e., the<?ry building, 

ought to be explicitly discussed. To fail to discuss specific ** 

theoretical propositions is to lose some of the value pfetheory's 

existence; this is especially* unfortunate since the theories do 

incorporate reasonably specific propositions about phenomena (see 

Davis andLSalasin, 1975, p. 641). A sample proposition is offered by 

Thomf^n (1971) who argued that communicators should "provide 

rationalizations for listeners who are unwilling to admit that 

« 

socially disapproved motives are responsible for their beliefs or 
actions' 1 (p. 185) - 

A second danger in simulation research is that some 
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•investigators will choose "satoples of convenience"' in their work 
•-€;Q«use of the .'constraints of administering the simulations to^ 
field-based" practitioners. It is unlikely that the business manors or 

» 1 ' * * ■ • 

education students who happen to enroll in graduate bourses are 

\ 

representative of the individuals ^fltfBom the researchers sometimes 
attempt to generalize* * This criticism is -independent of and probably 
more telling than tlfe recognition of sorire researchers (cf . Brown and 
Newman, 1979a, pp. 6-7) that slmuStion research may not perfectly 
.generalize to natural ecologies. . . * / 

| ** Third, simulation research is inherently *1 imited in that it. 

typically focuses on the reports phase of evaluation, an.d we know that 
I "what happens before the final report is written will usually • * 
Jdetermine' utilization" *(Patton, 1978, p. -266). Stiir/it must be 

acknowledged that if we ' really want' to, increase evaluatibn Jse, then 
jit may t be necessary to emphasise all phases of the evaluation 
'endeavor,, including those that may be relatively les's important in 
(determining* use * 

| s Weiss (1979) summarizes four\^pproaches to the study of use that 

differ <in their starting place's;^ i.e.\one^can trace the effects of 

I evaluation *y studying fep^ific evaluation ^tu)dies, by tracing the 
actions 6f specific peoplle, b% examining the /vari6us approaches to a 
[^iven issue, or by following the^effects of evaluations in a given 
.'organization (opt 16-17). To date, research WsT"&enterefc on the first 
'o£Hhe^\ >ppif6aches in that^botft naturalistic and simulation- studies 
(typically 'focus 'on the effects of one stud^ or one report.* Expanding 



^search to include' all £qur' types of studies should broaden the 
ejected results. + 
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/ m Research Agenda for the Future 

Despite some progress toward understanding use phenomena, much- 
remains to be learned. Several priorities can be identified for 

<■ 

ftftjjre research. 

,1. Further conceptual work is needed to clarify the definitions 

of' the terms use and rqfcionality in thp context of local 

decision-making involving evaluation information. Brown (1981) has 

suggested that 

We need a multi-dimensional definition of use whi^h 
embraces the questions of the kinds of use, who uses the 
information, what information is used, and the-context in 
which it is used, as well as the extent of usage (p. 7). 

2. Brown also notes that expectations of use could be made 

* added dimension of the* definition of 'use, but recommends that for now 

* they be, studied separately (p. 8). Conner (1979) has similarly 
suggested 'studying utilizatian goals and inputs \p. 17). 

3. Studies of the process of local evaluatfon use are needed, * 

and prospective case studies should be a high priority for. future 

research.- Conner (1979) makes this point, when he writes: 

The, absence of studies with a "current" time orientation is 
a serious missing link in the 'utilization research chain . , 
1 . I This orientat&mt is essential if we are to obtain the 
most accurate information about utilization. Retrospective 
studies, while useful, are subject to biases directly 
related to # .the type of use which has occurred (p.. 16). 

An assessment of the practicality of the collaborative approach to 

» 

evaluation is needed* . Brown (J.981) suggests the rreed^fSr a t 
delineation of t£e §teps involved in the process of use and 
"clarification of the decision-making (usage) context" (p. 8). Study 
should .also be made of the effetets of certain qutcomes (e.g., negative 
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results or conflicting results) provided at certain times and at 
certain stages of program development. Further/ it would be helpful 
_to examine the effects of the early identification of potential users 
and the selection of issues of concern to them (Weiss, 1972, p. -324). 

4. Additional -insight into how evaluators perceive themselves 
and are, perceived by administrators is needed if the person^ factor 
is accepted as an important determinant of use. A broader 
understanding of these perceptions is needed similar ±o that which has 
been achieved in areas such as teacher education (Mil\er, Thompson, 
and Frankiewicz, 1975)- 

5. Local use research needs to broaden its subjects to include 
more principals, Jieach'ecs, and 4 school board members. Several 

^researchers have ihvolv^ed persons in these roles as c subjects (e.g., * 
Kennedy, Apiing, and Neumann, 19B0 ; thompscnr, 1981; WilTiartfs and'Bank, 
1980)*, but the use of people in these roles as subjects is n^t \n : ' 
proportion to the influence that they exert over program operation. 
Regarding principals, for example, Lipham (1988) notes that' "th< 
leadership behavior of the principal is a powerful 'factor which 
influences the adoption, and institut ipn^Tiza'tion of an educational 
change" (py. 83) . $ 

6. Researchers need to determine whether or not school 

-r 

personnel can effectively be trained to make more optimal use of 
evaluative information. For example, research is needed to determine 
if it is feasible to help administrators increase "problem solving* 
capabilities and ability to express and articulate [information] 
needs" (Haenn, 1980*, p. 13) . Training of local evaluation staff 
'members should also be explored since "there is a strik ing \absence of. 

• " ' 72 ' . 
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any formal training in evaluation for staff (Lyon et al., 1978, p. 

7. ^ Given the importance of. informal communication, further ' * 
f empirical work, fs needed on the, effects of local dissemination ' 

efforts, As'Weisrs (1972) proposed, the following variables deserve 

examination: charity and attractiveness of presentation; inclusion of 

. ' . -V - ' ' ■ 

implications for action; use of inventive mechanisms to reach remote 
audiences;, and aggressive advocacy by evaluators (p. 325) . 

8, Research on the merfts of mandated evaluation is also 
needed* Do programs that .are not evaluated differ from comparable 
evaluated programs regarding either program processes or productivity? 
Also, at some point the. notion of mandated evaluation itself needs to 
be evaluated. 

Both Brown (1981) and; Sanders (1981) have suggested ' 8 

broadening the theoretical basis' of use research to include 
♦application of theories from other disciplines. The study of the 
literature on interpersonal influence, on communicat iontf theory, and I 
. -pn decision-making theory may well .provide further insight into the 



use or evaj/d^tive information. 

Making Local Evaluation /Use Happen . • 1 
People who discuss evaluation* use tend inappropriately to lay 
the blame for non-use at someone else*s* doorstep. However, everyone. 

4 , " 

involved in the evaluation-enterprise at the local Isvel must 
recognize that it takes at least two to create non-use. 
Administtators must assume some responsibility for making sure that 
\ evaluative infornfatip-n is both, usable and used (Meltsner,. 1976). ' So 
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too evaluators must accept some responsibility for making use happen. 
„ As Polivka and Steg (1978) argue, 

Traditionally, 'the evaluator has been very hesitant »to ^ 
claim any responsibility for the use of his/her -findings. 
This 'approach" has helped make it very easy to ignore 
' evaluation results (p. 697) . 

This* can no longer be the case. Standard A8, "Evaluation Impact" ' ' 

(Joint Committee, 1981), reminds us. that "evaluators must not assume 

. that improvements will occur automatically once the evaluation report 
is completed" (p. 47) . Both administrators and evaluators need to 
recognize that the responsibility for use is not a "zero-sjum" game in 
which responsibility can be divided up and will always total 100%. 
They would be better* off if, both administrators and evaluators assumed 
60 or 8t)% of the responsibility fcor increasing use? Even then, as 

. Patton (1978), notes, "Increasing utilization potential does not 
guarantee utilization of findings. There are no guarantees" (p, 96). 
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